Oct 30 21:45:47.860248 ip-10-0-133-91 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:45:48.218806 ip-10-0-133-91 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:45:48.218806 ip-10-0-133-91 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:45:48.218806 ip-10-0-133-91 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:45:48.218806 ip-10-0-133-91 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:45:48.222431 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222364 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.222431 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222432 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222438 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222442 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222445 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222448 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222451 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222457 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222460 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222462 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222465 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222468 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222471 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222473 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222482 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222485 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222488 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222490 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.222497 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222493 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222497 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222501 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222504 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222507 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222510 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222513 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222515 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222518 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222520 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222524 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222528 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222530 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222533 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222536 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222538 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222541 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222543 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222546 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.222904 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222548 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222551 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222554 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222557 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222560 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222562 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222565 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222567 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222569 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222572 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222575 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222577 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222580 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222583 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222586 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222589 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222591 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222594 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222596 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222599 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.223357 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222601 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222604 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222606 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222609 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222611 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222614 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222616 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222619 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222621 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222624 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222626 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222629 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222631 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222634 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222636 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222639 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222642 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222647 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222649 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.223820 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222651 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222654 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222656 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222659 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222662 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222665 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222668 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222670 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222673 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222675 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222678 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222681 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222684 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222687 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222689 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222692 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222694 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222696 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222699 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.224304 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222701 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222703 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222706 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222708 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222710 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222713 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222716 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222720 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.222724 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223464 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223470 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223473 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223476 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223479 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223482 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223485 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223487 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223490 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223493 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223496 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.224779 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223499 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223501 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223504 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223506 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223509 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223512 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223514 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223516 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223519 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223521 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223524 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223526 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223529 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223531 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223534 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223536 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223539 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223541 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223544 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.225271 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223546 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223548 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223551 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223554 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223557 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223559 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223561 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223564 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223566 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223569 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223572 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223574 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223577 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223580 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223582 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223584 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223587 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223589 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223592 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223594 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.225723 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223596 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223599 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223601 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223604 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223606 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223608 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223611 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223613 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223616 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223618 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223621 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223623 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223626 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223630 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223633 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223637 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223640 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223642 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223645 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223647 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.226205 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223650 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223652 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223655 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223658 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223661 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223663 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223671 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223673 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223676 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223678 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223681 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223683 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223686 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223688 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223691 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223693 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223696 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223698 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223701 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.226692 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223703 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223709 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223711 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223713 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223716 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223718 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223720 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223724 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223728 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223732 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223735 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223738 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223741 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223743 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223746 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.223749 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224232 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224238 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.227181 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224240 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224243 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224246 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224249 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224251 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224254 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224256 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224261 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224264 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224266 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224269 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224271 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224274 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224276 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224279 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224282 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224284 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224289 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.227614 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224292 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224297 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224301 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224305 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224308 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224311 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224314 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224317 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224319 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224322 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224324 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224327 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224329 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224332 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224334 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224337 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224339 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224342 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224344 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.228121 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224347 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224349 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224351 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224354 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224356 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224359 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224361 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224364 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224366 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224369 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224372 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224377 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224379 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224382 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224384 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224388 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224390 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224394 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.228624 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224399 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224401 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224404 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224407 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224410 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224412 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224414 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224417 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224419 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224422 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224424 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224427 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224429 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224431 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224434 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224437 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224439 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224442 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224444 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224446 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.229109 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224449 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224451 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224454 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224456 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224458 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224461 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224465 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224468 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224471 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224473 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224476 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224479 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224481 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224484 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224486 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224489 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224491 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224494 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224497 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.229577 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224499 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224501 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224504 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224506 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224509 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224512 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.224514 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225416 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225426 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225432 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225436 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225441 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225444 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225448 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225452 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225455 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225458 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225461 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225464 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225469 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225472 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225475 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225478 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:45:48.230054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225481 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225485 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225489 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225492 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225495 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225499 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225503 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225506 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225509 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225512 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225515 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225518 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225521 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225524 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225528 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225530 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225535 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225537 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225540 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225543 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225546 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225549 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225553 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225556 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 21:45:48.230584 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225558 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225562 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225565 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225569 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225575 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225578 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225581 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225583 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225586 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225589 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225592 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225595 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225598 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225602 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225609 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225612 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225615 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225618 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225621 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225624 2578 flags.go:64] FLAG: --help="false" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225626 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225630 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225633 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:45:48.231261 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225635 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225639 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225642 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225645 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225647 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225650 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225653 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225656 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225659 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225662 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225665 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225667 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225670 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225675 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225677 2578 flags.go:64] FLAG: --lock-file="" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225680 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225683 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225686 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225691 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225694 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225697 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225700 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225704 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225707 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:45:48.231795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225710 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225713 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225717 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225720 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225724 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225727 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225730 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225733 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225737 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225740 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225742 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225745 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225752 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225755 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225758 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225761 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225764 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225767 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225770 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225772 2578 flags.go:64] FLAG: --port="10250" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225775 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225780 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0db8354240741ede1" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225783 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225786 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225789 2578 flags.go:64] FLAG: --register-node="true" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225792 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:45:48.232389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225796 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225799 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225801 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225806 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225811 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225814 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225817 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225820 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225823 2578 flags.go:64] FLAG: --runonce="false" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225825 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225828 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225831 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225834 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225836 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225839 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225842 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225845 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225848 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225851 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225854 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225856 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225859 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225862 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225865 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225870 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:45:48.233009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225873 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225875 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225880 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225883 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225886 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225889 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225892 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225894 2578 flags.go:64] FLAG: --v="2" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225898 2578 flags.go:64] FLAG: --version="false" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225903 2578 flags.go:64] FLAG: --vmodule="" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225907 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.225911 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226053 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226058 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226061 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226064 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226066 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226069 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226072 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226074 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226077 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226080 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.233591 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226082 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226085 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226088 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226091 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226093 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226096 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226098 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226100 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226103 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226105 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226108 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226111 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226114 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226117 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226119 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226122 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226124 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226127 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226129 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.234123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226133 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226137 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226140 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226144 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226147 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226150 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226152 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226155 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226157 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226160 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226163 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226165 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226168 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226170 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226173 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226175 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226178 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226180 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226183 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226185 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.234572 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226188 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226190 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226193 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226195 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226198 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226201 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226204 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226206 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226209 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226212 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226214 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226220 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226222 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226225 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226227 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226230 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226233 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226235 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226238 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.235123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226240 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226242 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226245 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226248 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226250 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226253 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226255 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226258 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226260 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226262 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226265 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226268 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226270 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226272 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226275 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226277 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226280 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226283 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226285 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226288 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.235585 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226290 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226293 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226295 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226298 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226302 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226304 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226307 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226309 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226312 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226314 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226317 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226319 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226324 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226328 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226332 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.236193 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226335 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.226361 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226394 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226399 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226402 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226405 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226407 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226410 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226413 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226415 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226418 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226421 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226432 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226437 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226441 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226443 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.236575 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226446 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226448 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226451 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226453 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226458 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226460 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226463 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226465 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226468 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226470 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226474 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226476 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226479 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226481 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226484 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226487 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226489 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226492 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226495 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.236990 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226498 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226500 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226503 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226505 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226507 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226510 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226512 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226514 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226517 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226520 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226522 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226526 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226528 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226531 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226533 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226535 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226538 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226542 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226544 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226546 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.237491 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226549 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226551 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226554 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226557 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226560 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226562 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226565 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226567 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226570 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226572 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226575 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226579 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226582 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226584 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226587 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226589 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226592 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226594 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.238226 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226596 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226599 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226601 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226604 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226607 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226610 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226612 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226615 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226617 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226619 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226623 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226627 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226630 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226634 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226636 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226639 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226641 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226644 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226648 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.238680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226650 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226653 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226655 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226658 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226660 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226663 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226665 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226667 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226670 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226672 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226675 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226677 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226679 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226682 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.226684 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.233011 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.233030 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.233046 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.233050 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:45:48.239172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233110 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233116 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233119 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233122 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233124 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233127 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233129 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233132 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233134 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233138 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233140 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233142 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233145 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233148 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233150 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233153 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233155 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233158 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233161 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233163 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.239636 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233166 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233168 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233170 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233173 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233175 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233178 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233181 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233183 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233186 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233195 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233198 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233200 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233202 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233205 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233208 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233210 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233214 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233218 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233221 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.240123 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233224 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233227 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233231 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233235 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233238 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233241 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233244 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233246 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233249 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233251 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233254 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233256 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233259 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233261 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233263 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233265 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233268 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233270 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233273 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.240570 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233276 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233278 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233281 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233284 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233296 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233301 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233303 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233306 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233311 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233313 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233315 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233318 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233320 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233322 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233325 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233327 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233329 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.241320 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233332 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233334 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233337 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233340 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233343 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233345 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233347 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233350 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233353 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233355 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233358 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233363 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233366 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233368 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233371 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233374 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233383 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233385 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233388 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.242086 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233390 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233393 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233395 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233398 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233401 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233403 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233405 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233408 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.233434 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233614 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233619 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233622 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233625 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233628 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233630 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.242680 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233633 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233635 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233638 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233641 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233644 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233646 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233649 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233651 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233654 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233656 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233659 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233661 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233664 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233666 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233669 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233677 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233680 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233682 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233685 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.243200 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233687 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233690 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233692 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233695 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233697 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233700 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233702 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233705 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233707 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233710 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233712 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233715 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233718 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233721 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233723 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233725 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233728 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233730 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233733 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.243665 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233735 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233737 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233740 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233742 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233745 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233747 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233750 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233752 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233754 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233757 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233764 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233767 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233769 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233772 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233774 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233777 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233779 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233781 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233784 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233786 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.244282 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233790 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233793 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233796 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233798 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233801 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233804 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233806 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233809 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233814 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233816 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233819 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233821 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233824 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233826 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233829 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233831 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233834 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233836 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233838 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.244777 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233841 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233843 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233845 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233854 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233856 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233859 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233861 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233864 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233866 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233870 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233874 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233876 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233879 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233881 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233884 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233886 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233888 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233891 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233894 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.245311 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233896 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233898 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233901 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.233923 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233984 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233990 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233992 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233995 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.233997 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234000 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234002 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234005 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234007 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234010 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234012 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234014 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:45:48.245782 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234017 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234025 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234028 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234031 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234033 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234036 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234038 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234040 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234043 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234045 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234047 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234050 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234053 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234055 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234058 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234060 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234062 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234065 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234068 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:45:48.246172 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234070 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234072 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234075 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234077 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234079 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234082 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234084 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234087 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234089 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234092 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234094 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234096 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234099 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234101 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234103 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234111 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234117 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234119 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234121 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:45:48.246615 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234124 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234126 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234130 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234133 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234137 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234139 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234142 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234144 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234147 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234150 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234153 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234155 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234157 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234160 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234162 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234165 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234167 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234169 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234172 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234174 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:45:48.247136 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234177 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234179 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234181 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234184 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234186 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234188 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234191 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234193 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234201 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234204 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234207 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234211 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234214 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234217 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234220 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234223 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234225 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234228 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234230 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:45:48.247580 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234233 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234236 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234238 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234240 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234244 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234247 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234249 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234252 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234255 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234258 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234260 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234262 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234265 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234267 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234269 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:48.234272 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.234337 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.237080 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.237895 2578 server.go:1014] "Starting client certificate rotation" Oct 30 21:45:48.248019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.238017 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:45:48.248443 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.238833 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:45:48.248443 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.242241 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:45:48.248443 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.245122 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:45:48.256911 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.256893 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:45:48.260272 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.260257 2578 log.go:25] "Validated CRI v1 image API" Oct 30 21:45:48.261065 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.261041 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:45:48.263490 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.263467 2578 fs.go:135] Filesystem UUIDs: map[26f7696d-b143-43db-ae3f-ca776eea8b11:/dev/nvme0n1p4 708024d4-dc90-4ff5-b922-57cad575a694:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:45:48.263559 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.263489 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:45:48.267532 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.267431 2578 manager.go:217] Machine: {Timestamp:2026-10-30 21:45:48.266967741 +0000 UTC m=+0.301536289 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122661 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25096ae0b08b28544b251e8de6ab5d SystemUUID:ec25096a-e0b0-8b28-544b-251e8de6ab5d BootID:d8ba97bb-01b3-4da1-928c-7865669161ff Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f7:f2:00:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f7:f2:00:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:e0:34:7d:98:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:45:48.267532 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.267528 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:45:48.267632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.267615 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:45:48.270128 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.270105 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:45:48.270252 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.270129 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-91.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:45:48.270297 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.270254 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:45:48.270297 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.270261 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:45:48.270297 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.270271 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:45:48.270297 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.270280 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:45:48.271019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.271008 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:45:48.274752 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.274740 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:45:48.274855 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.274846 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:45:48.275585 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.275576 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:45:48.275614 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.275590 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:45:48.275614 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.275600 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:45:48.275614 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.275608 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:45:48.275707 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.275616 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:45:48.275872 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.275860 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:45:48.277867 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.277848 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:45:48.282514 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.282498 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:45:48.283269 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283256 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:45:48.283310 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283294 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:45:48.283453 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283439 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:45:48.283453 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283454 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:45:48.283563 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283461 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:45:48.283563 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283466 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:45:48.283563 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283471 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:45:48.283563 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283477 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:45:48.283944 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283935 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:45:48.283989 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283944 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:45:48.283989 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283969 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:45:48.283989 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283976 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:45:48.283989 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283989 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:45:48.284142 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.283998 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:45:48.284689 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.284676 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:45:48.284720 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.284692 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:45:48.288232 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.288218 2578 server.go:1263] "Started kubelet" Oct 30 21:45:48.288305 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.288275 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:45:48.288386 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.288339 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:45:48.291592 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.291572 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:45:48.292082 ip-10-0-133-91 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:45:48.293150 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.292931 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:45:48.293811 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.293792 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:45:48.297103 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.297085 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:45:48.299018 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.299001 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:45:48.299816 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.299800 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:45:48.299896 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.299876 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" Oct 30 21:45:48.300334 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300312 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:45:48.300445 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300424 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:45:48.300445 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300446 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:45:48.300815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300526 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:45:48.300815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300548 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:45:48.300815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300608 2578 factory.go:55] Registering systemd factory Oct 30 21:45:48.300815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300620 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:45:48.300815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300804 2578 factory.go:153] Registering CRI-O factory Oct 30 21:45:48.300815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300816 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 21:45:48.301116 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300852 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:45:48.301116 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300865 2578 factory.go:103] Registering Raw factory Oct 30 21:45:48.301116 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.300875 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 21:45:48.301750 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.301735 2578 manager.go:319] Starting recovery of all containers Oct 30 21:45:48.313485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.313472 2578 manager.go:324] Recovery completed Oct 30 21:45:48.317279 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.317267 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.320620 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.320594 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.320721 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.320626 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.320721 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.320636 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.321171 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.321148 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:45:48.321254 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.321172 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:45:48.321254 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.321156 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.321254 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.321191 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:45:48.323289 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.323269 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.323386 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.323306 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.323386 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.323321 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.323568 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.323520 2578 policy_none.go:50] "Start" Oct 30 21:45:48.323568 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.323532 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:45:48.323568 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.323544 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:45:48.325676 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.325665 2578 policy_none.go:44] "Start" Oct 30 21:45:48.366498 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.366470 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:45:48.368678 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.368667 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 21:45:48.369102 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.369084 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:45:48.369155 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.369105 2578 server.go:87] "Starting device plugin registration server" Oct 30 21:45:48.369357 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.369346 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:45:48.369395 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.369361 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:45:48.369494 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.369476 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:45:48.369576 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.369565 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:45:48.369629 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.369576 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:45:48.369939 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.369922 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:45:48.370025 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.369984 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" Oct 30 21:45:48.436091 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.436056 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:45:48.437340 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.437322 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:45:48.437404 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.437352 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:45:48.437404 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.437375 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:45:48.437485 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.437413 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:45:48.440459 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.440443 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-91.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:45:48.440581 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.440514 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-91.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:45:48.441390 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.440434 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-91.us-west-2.compute.internal.18e36cd73b0eddf6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-91.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-91.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:48.288187894 +0000 UTC m=+0.322756440,LastTimestamp:2026-10-30 21:45:48.288187894 +0000 UTC m=+0.322756440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-91.us-west-2.compute.internal,}" Oct 30 21:45:48.470393 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.470355 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.471141 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.471123 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.471204 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.471155 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.471204 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.471165 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.471204 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.471185 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.510030 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.510011 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:45:48.510088 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.510077 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v5wdc" Oct 30 21:45:48.537858 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.537839 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal"] Oct 30 21:45:48.537904 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.537897 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.538683 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.538664 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.538747 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.538691 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.538747 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.538703 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.540873 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.540861 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.541037 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.541075 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541049 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.541783 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541769 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.541849 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541797 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.541849 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541807 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.541947 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541774 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.541947 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541878 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.541947 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.541897 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.544036 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.544022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.544098 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.544049 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:45:48.544787 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.544768 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:45:48.544872 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.544800 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:45:48.544872 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.544814 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:45:48.570399 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.570382 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.576806 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.576789 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.595906 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.595893 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:48.597256 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.597243 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:48.602473 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.602454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4385e5f7b5de508595fd215a6736486a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"4385e5f7b5de508595fd215a6736486a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.602552 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.602486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.602552 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.602511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.689737 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.689708 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:48.692020 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.691999 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:48.692840 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.692823 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.692889 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:48.692850 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-91.us-west-2.compute.internal\": node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" Oct 30 21:45:48.701007 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.700991 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.703158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.703142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4385e5f7b5de508595fd215a6736486a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"4385e5f7b5de508595fd215a6736486a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.703201 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.703168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.703201 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.703186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.703259 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.703241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.703259 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.703248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4385e5f7b5de508595fd215a6736486a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"4385e5f7b5de508595fd215a6736486a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.703318 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.703257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.762410 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.762246 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:45:48.763039 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.763026 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.825650 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.825626 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:45:48.873267 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.873247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:48.879755 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:48.879733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:45:49.238222 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.238165 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:45:49.238665 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.238335 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:45:49.238665 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.238365 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:45:49.238665 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.238339 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a26314a26c7f3463b9bcb9cfe0458715-c9cd20b13308b9a1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-91.us-west-2.compute.internal.18e36cd73cfe0af7\": read tcp 10.0.133.91:37354->18.225.12.250:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-91.us-west-2.compute.internal.18e36cd73cfe0af7 default 5097 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-91.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-91.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-91.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:48 +0000 UTC,LastTimestamp:2026-10-30 21:45:48.471169657 +0000 UTC m=+0.505738203,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-91.us-west-2.compute.internal,}" Oct 30 21:45:49.276151 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.276132 2578 apiserver.go:52] "Watching apiserver" Oct 30 21:45:49.299236 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.299216 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:45:49.515228 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.515155 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:40:49 +0000 UTC" deadline="2028-05-23 06:19:56.782140269 +0000 UTC" Oct 30 21:45:49.515228 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.515188 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13688h34m7.266955241s" Oct 30 21:45:49.557968 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.557921 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:45:49.558314 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.558290 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-pj2jw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal","openshift-multus/multus-bvk6g","kube-system/konnectivity-agent-dql2n","openshift-multus/network-metrics-daemon-7lq8c","kube-system/global-pull-secret-syncer-687l6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n","openshift-dns/node-resolver-92scn","openshift-multus/multus-additional-cni-plugins-8xjbc","openshift-image-registry/node-ca-t8l67","openshift-network-diagnostics/network-check-target-zbzd4","openshift-network-operator/iptables-alerter-gqkkp","openshift-ovn-kubernetes/ovnkube-node-2s925"] Oct 30 21:45:49.560815 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.560789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.562892 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.562863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.565026 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.565006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.567270 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.567248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:49.567355 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.567331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:45:49.569444 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.569428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.569504 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.569484 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:45:49.571623 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.571608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.573795 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.573780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.576129 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.576108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.578479 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.578460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.580561 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.580536 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:49.580668 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.580602 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:45:49.582724 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.582705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.585141 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.585119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.601783 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.601754 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:45:49.608075 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-sys-fs\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.608160 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cw9d\" (UniqueName: \"kubernetes.io/projected/285863f3-fa2b-4d9a-80b2-7421899e97c4-kube-api-access-2cw9d\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.608160 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-etc-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608160 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-ovn\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608160 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-node-log\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608321 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-log-socket\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608321 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.608321 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-lib-modules\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.608321 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.608441 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-device-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.608441 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.608524 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-systemd-units\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608524 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608524 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-config\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729a10ce-45d9-4094-8f2b-c5af2b19d159-agent-certs\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.608612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-host-slash\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.608612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-socket-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.608612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-registration-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.608736 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.608736 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/285863f3-fa2b-4d9a-80b2-7421899e97c4-host\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.608736 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-var-lib-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608736 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-cni-binary-copy\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.608736 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-slash\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-netns\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-etc-kubernetes\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-conf\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-host\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgds\" (UniqueName: \"kubernetes.io/projected/2e9328b7-cd5f-4a63-8895-be053051f188-kube-api-access-nkgds\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-systemd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.608930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-bin\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-script-lib\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.608998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysconfig\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-kubelet\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-k8s-cni-cncf-io\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kb5\" (UniqueName: \"kubernetes.io/projected/191e72fd-de5a-4dea-9592-c75367da8608-kube-api-access-55kb5\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qt92\" (UniqueName: \"kubernetes.io/projected/35cd7059-12c6-4c41-99ef-86eea5a5873e-kube-api-access-8qt92\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-kubernetes\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.609170 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-etc-tuned\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-run\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-bin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-multus-certs\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-kubelet-config\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75sw\" (UniqueName: \"kubernetes.io/projected/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-kube-api-access-k75sw\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-var-lib-kubelet\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-system-cni-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsts\" (UniqueName: \"kubernetes.io/projected/248fb46a-f3b1-4d89-9d88-0191280d983d-kube-api-access-pjsts\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-netd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovn-node-metrics-cert\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-systemd\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8639240c-cb7f-45b0-a0ce-97eb850716cc-tmp-dir\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhtv\" (UniqueName: \"kubernetes.io/projected/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-kube-api-access-5dhtv\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cd9p\" (UniqueName: \"kubernetes.io/projected/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kube-api-access-5cd9p\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhxz\" (UniqueName: \"kubernetes.io/projected/8639240c-cb7f-45b0-a0ce-97eb850716cc-kube-api-access-skhxz\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-env-overrides\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-multus\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-kubelet\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.609731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-sys\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-cnibin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-netns\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-hostroot\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-conf-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-dbus\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8639240c-cb7f-45b0-a0ce-97eb850716cc-hosts-file\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.609982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-multus-daemon-config\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729a10ce-45d9-4094-8f2b-c5af2b19d159-konnectivity-ca\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/285863f3-fa2b-4d9a-80b2-7421899e97c4-serviceca\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-iptables-alerter-script\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.610147 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-tmp\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.610632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-socket-dir-parent\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.610632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-modprobe-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.610632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-cnibin\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.610632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.610225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-system-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.624324 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.624299 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:45:49.624475 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.624458 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kjn6x\"" Oct 30 21:45:49.624551 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.624535 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fd5qx\"" Oct 30 21:45:49.624606 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.624587 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5wwct\"" Oct 30 21:45:49.629545 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.629522 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mkvf\"" Oct 30 21:45:49.629650 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.629545 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:45:49.635027 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635000 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqqg2\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635047 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635048 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635087 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635070 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635088 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635086 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635115 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.635158 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635158 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.635425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635169 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:45:49.635425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635200 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.635425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635202 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.635425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635214 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.635425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635225 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:45:49.635425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.635215 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rn9hr\"" Oct 30 21:45:49.662751 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.662729 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:45:49.662751 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.662736 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.665535 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.665517 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:45:49.665596 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.665560 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bg2hw\"" Oct 30 21:45:49.665596 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.665565 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.674011 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.673994 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:45:49.674120 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.674042 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9dfpb\"" Oct 30 21:45:49.674120 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.674051 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.674243 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.674234 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:45:49.674912 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.674888 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.675067 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.675052 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:45:49.675151 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.675072 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:45:49.675151 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.675109 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:45:49.675151 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.675122 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:45:49.675151 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.675123 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvjtk\"" Oct 30 21:45:49.676544 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.676528 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:45:49.676633 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.676560 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8x4g" Oct 30 21:45:49.711126 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-k8s-cni-cncf-io\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711200 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kb5\" (UniqueName: \"kubernetes.io/projected/191e72fd-de5a-4dea-9592-c75367da8608-kube-api-access-55kb5\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711200 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qt92\" (UniqueName: \"kubernetes.io/projected/35cd7059-12c6-4c41-99ef-86eea5a5873e-kube-api-access-8qt92\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:49.711200 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-kubernetes\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711200 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-k8s-cni-cncf-io\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-etc-tuned\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-run\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-kubernetes\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-bin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711331 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-bin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-multus-certs\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-kubelet-config\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.711408 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-run\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75sw\" (UniqueName: \"kubernetes.io/projected/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-kube-api-access-k75sw\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-multus-certs\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-kubelet-config\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-var-lib-kubelet\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-system-cni-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsts\" (UniqueName: \"kubernetes.io/projected/248fb46a-f3b1-4d89-9d88-0191280d983d-kube-api-access-pjsts\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-var-lib-kubelet\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-netd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovn-node-metrics-cert\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-system-cni-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-systemd\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-netd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8639240c-cb7f-45b0-a0ce-97eb850716cc-tmp-dir\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-systemd\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhtv\" (UniqueName: \"kubernetes.io/projected/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-kube-api-access-5dhtv\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.711913 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cd9p\" (UniqueName: \"kubernetes.io/projected/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kube-api-access-5cd9p\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711736 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhxz\" (UniqueName: \"kubernetes.io/projected/8639240c-cb7f-45b0-a0ce-97eb850716cc-kube-api-access-skhxz\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-env-overrides\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-multus\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-kubelet\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711924 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8639240c-cb7f-45b0-a0ce-97eb850716cc-tmp-dir\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.711972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-sys\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-cnibin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-netns\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.712078 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.712126 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.712140 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:50.212118668 +0000 UTC m=+2.246687215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:49.712695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-hostroot\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.712180 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs podName:35cd7059-12c6-4c41-99ef-86eea5a5873e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:50.212164103 +0000 UTC m=+2.246732641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs") pod "network-metrics-daemon-7lq8c" (UID: "35cd7059-12c6-4c41-99ef-86eea5a5873e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712189 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-hostroot\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-kubelet\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-multus\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-netns\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-sys\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712312 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-cnibin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-conf-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-dbus\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8639240c-cb7f-45b0-a0ce-97eb850716cc-hosts-file\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-conf-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712482 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-dbus\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8639240c-cb7f-45b0-a0ce-97eb850716cc-hosts-file\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.713483 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-multus-daemon-config\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729a10ce-45d9-4094-8f2b-c5af2b19d159-konnectivity-ca\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/285863f3-fa2b-4d9a-80b2-7421899e97c4-serviceca\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-iptables-alerter-script\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-tmp\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-socket-dir-parent\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-modprobe-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-cnibin\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-system-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-sys-fs\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cw9d\" (UniqueName: \"kubernetes.io/projected/285863f3-fa2b-4d9a-80b2-7421899e97c4-kube-api-access-2cw9d\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-etc-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-ovn\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-env-overrides\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.713374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-node-log\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714251 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.713629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-multus-daemon-config\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.712932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-node-log\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-log-socket\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-iptables-alerter-script\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-lib-modules\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-device-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-systemd-units\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-config\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729a10ce-45d9-4094-8f2b-c5af2b19d159-agent-certs\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-host-slash\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-socket-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-registration-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.714943 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/285863f3-fa2b-4d9a-80b2-7421899e97c4-host\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-var-lib-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-cni-binary-copy\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-slash\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-netns\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-etc-kubernetes\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-conf\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-host\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.714989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgds\" (UniqueName: \"kubernetes.io/projected/2e9328b7-cd5f-4a63-8895-be053051f188-kube-api-access-nkgds\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-systemd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-bin\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-script-lib\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysconfig\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-kubelet\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.715708 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/285863f3-fa2b-4d9a-80b2-7421899e97c4-serviceca\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715801 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-etc-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-ovn\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.715918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovn-node-metrics-cert\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-slash\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716169 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-log-socket\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-etc-kubernetes\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716222 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-netns\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-conf\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-cni-binary-copy\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.716485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.717024 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729a10ce-45d9-4094-8f2b-c5af2b19d159-konnectivity-ca\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.717024 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.716533 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-device-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.717358 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.717332 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/285863f3-fa2b-4d9a-80b2-7421899e97c4-host\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.717358 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.717342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-sys-fs\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.717591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.717570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.717825 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.717803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-bin\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.718033 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.718015 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysconfig\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.718193 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.718176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-kubelet\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.718336 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.718319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-system-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.718485 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.718468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-socket-dir-parent\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.718661 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.718595 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-modprobe-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.718758 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.718677 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-systemd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.719138 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-host\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.719229 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-systemd-units\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.719285 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-registration-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.719343 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-host-slash\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.719343 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-cnibin\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.719653 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-var-lib-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.719763 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-lib-modules\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.719763 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.719734 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.720215 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720174 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-config\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.720215 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-socket-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.720392 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-etc-tuned\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.720458 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.720458 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-script-lib\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.720458 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-tmp\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.720573 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.720486 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729a10ce-45d9-4094-8f2b-c5af2b19d159-agent-certs\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.780398 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.780343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75sw\" (UniqueName: \"kubernetes.io/projected/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-kube-api-access-k75sw\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:49.780710 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.780692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cw9d\" (UniqueName: \"kubernetes.io/projected/285863f3-fa2b-4d9a-80b2-7421899e97c4-kube-api-access-2cw9d\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.782355 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.782331 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qt92\" (UniqueName: \"kubernetes.io/projected/35cd7059-12c6-4c41-99ef-86eea5a5873e-kube-api-access-8qt92\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:49.782819 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.782800 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhxz\" (UniqueName: \"kubernetes.io/projected/8639240c-cb7f-45b0-a0ce-97eb850716cc-kube-api-access-skhxz\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.782930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.782914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kb5\" (UniqueName: \"kubernetes.io/projected/191e72fd-de5a-4dea-9592-c75367da8608-kube-api-access-55kb5\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.783519 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.783499 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:49.783519 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.783520 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:49.783655 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.783532 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxksq for pod openshift-network-diagnostics/network-check-target-zbzd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:49.783655 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:49.783610 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq podName:90fc0a23-c03b-40f3-83ac-61ba7b26bc5e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:50.283593826 +0000 UTC m=+2.318162378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hxksq" (UniqueName: "kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq") pod "network-check-target-zbzd4" (UID: "90fc0a23-c03b-40f3-83ac-61ba7b26bc5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:49.785695 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.785670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cd9p\" (UniqueName: \"kubernetes.io/projected/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kube-api-access-5cd9p\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.785802 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.785785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsts\" (UniqueName: \"kubernetes.io/projected/248fb46a-f3b1-4d89-9d88-0191280d983d-kube-api-access-pjsts\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.787835 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.787813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgds\" (UniqueName: \"kubernetes.io/projected/2e9328b7-cd5f-4a63-8895-be053051f188-kube-api-access-nkgds\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.789038 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.789023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhtv\" (UniqueName: \"kubernetes.io/projected/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-kube-api-access-5dhtv\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:49.898636 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.898615 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:45:49.916109 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.916079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bvk6g" Oct 30 21:45:49.935301 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.935278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:49.953058 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.953034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:45:49.966898 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.966878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92scn" Oct 30 21:45:49.984380 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.984365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:45:49.987013 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.986999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:45:49.990453 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:49.990439 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:45:50.029839 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.029823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:45:50.039564 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.039530 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4385e5f7b5de508595fd215a6736486a.slice/crio-2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103 WatchSource:0}: Error finding container 2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103: Status 404 returned error can't find the container with id 2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103 Oct 30 21:45:50.039979 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.039935 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bae0c6b7f3f9e1322bca7f9070d553.slice/crio-5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab WatchSource:0}: Error finding container 5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab: Status 404 returned error can't find the container with id 5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab Oct 30 21:45:50.040650 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.040625 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cf0a1a_efb8_4aef_ac2a_16e294018ccf.slice/crio-9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5 WatchSource:0}: Error finding container 9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5: Status 404 returned error can't find the container with id 9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5 Oct 30 21:45:50.045114 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.045095 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:45:50.219726 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.219706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:50.219833 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.219756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:50.219895 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.219847 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:50.219895 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.219858 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:50.219895 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.219893 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs podName:35cd7059-12c6-4c41-99ef-86eea5a5873e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:51.219877465 +0000 UTC m=+3.254446001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs") pod "network-metrics-daemon-7lq8c" (UID: "35cd7059-12c6-4c41-99ef-86eea5a5873e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:50.220032 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.219919 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:51.219899919 +0000 UTC m=+3.254468487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:50.319982 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.319936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:50.320468 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.320095 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:50.320468 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.320113 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:50.320468 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.320124 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxksq for pod openshift-network-diagnostics/network-check-target-zbzd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:50.320468 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:50.320177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq podName:90fc0a23-c03b-40f3-83ac-61ba7b26bc5e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:51.320160045 +0000 UTC m=+3.354728594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxksq" (UniqueName: "kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq") pod "network-check-target-zbzd4" (UID: "90fc0a23-c03b-40f3-83ac-61ba7b26bc5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:50.441340 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.441298 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5"} Oct 30 21:45:50.442422 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.442402 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"f4bae0c6b7f3f9e1322bca7f9070d553","Type":"ContainerStarted","Data":"5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab"} Oct 30 21:45:50.443367 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.443348 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"4385e5f7b5de508595fd215a6736486a","Type":"ContainerStarted","Data":"2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103"} Oct 30 21:45:50.475731 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.475710 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:50.680386 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.680332 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:50 +0000 UTC" deadline="2028-07-27 23:41:10.510261872 +0000 UTC" Oct 30 21:45:50.680386 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.680355 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15265h55m19.829909223s" Oct 30 21:45:50.774177 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.774154 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9328b7_cd5f_4a63_8895_be053051f188.slice/crio-6435f0bb8955515bf297cd59fc4ecb0890968609603be7fd0cf28b9c1e5a77f5 WatchSource:0}: Error finding container 6435f0bb8955515bf297cd59fc4ecb0890968609603be7fd0cf28b9c1e5a77f5: Status 404 returned error can't find the container with id 6435f0bb8955515bf297cd59fc4ecb0890968609603be7fd0cf28b9c1e5a77f5 Oct 30 21:45:50.809047 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.809007 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ba5b0e_7fee_4e39_a4bf_a6ac1e9c488a.slice/crio-54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b WatchSource:0}: Error finding container 54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b: Status 404 returned error can't find the container with id 54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b Oct 30 21:45:50.819352 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:50.819334 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:45:50.835462 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.835442 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729a10ce_45d9_4094_8f2b_c5af2b19d159.slice/crio-5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487 WatchSource:0}: Error finding container 5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487: Status 404 returned error can't find the container with id 5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487 Oct 30 21:45:50.889855 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.889832 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285863f3_fa2b_4d9a_80b2_7421899e97c4.slice/crio-c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31 WatchSource:0}: Error finding container c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31: Status 404 returned error can't find the container with id c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31 Oct 30 21:45:50.924755 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.924733 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191e72fd_de5a_4dea_9592_c75367da8608.slice/crio-a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2 WatchSource:0}: Error finding container a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2: Status 404 returned error can't find the container with id a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2 Oct 30 21:45:50.925476 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:50.925449 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8639240c_cb7f_45b0_a0ce_97eb850716cc.slice/crio-8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba WatchSource:0}: Error finding container 8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba: Status 404 returned error can't find the container with id 8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba Oct 30 21:45:51.033901 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:51.033877 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248fb46a_f3b1_4d89_9d88_0191280d983d.slice/crio-a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03 WatchSource:0}: Error finding container a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03: Status 404 returned error can't find the container with id a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03 Oct 30 21:45:51.078377 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:45:51.078355 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb385ed2_d5f0_4025_b614_1db5e6f0dd68.slice/crio-7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9 WatchSource:0}: Error finding container 7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9: Status 404 returned error can't find the container with id 7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9 Oct 30 21:45:51.226595 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.226550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:51.226735 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.226612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:51.226735 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.226727 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:51.226857 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.226795 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:53.226775174 +0000 UTC m=+5.261343721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:51.226857 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.226731 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:51.226857 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.226844 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs podName:35cd7059-12c6-4c41-99ef-86eea5a5873e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:53.226831945 +0000 UTC m=+5.261400495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs") pod "network-metrics-daemon-7lq8c" (UID: "35cd7059-12c6-4c41-99ef-86eea5a5873e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:51.327463 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.327392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:51.328018 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.327572 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:51.328018 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.327591 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:51.328018 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.327604 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxksq for pod openshift-network-diagnostics/network-check-target-zbzd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:51.328018 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.327659 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq podName:90fc0a23-c03b-40f3-83ac-61ba7b26bc5e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:53.327641697 +0000 UTC m=+5.362210236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxksq" (UniqueName: "kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq") pod "network-check-target-zbzd4" (UID: "90fc0a23-c03b-40f3-83ac-61ba7b26bc5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:51.438244 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.438202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:51.438443 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.438338 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:45:51.438443 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.438429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:51.438570 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.438512 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:45:51.438630 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.438579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:51.438684 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:51.438664 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:45:51.447700 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.447666 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerStarted","Data":"a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03"} Oct 30 21:45:51.448932 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.448901 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92scn" event={"ID":"8639240c-cb7f-45b0-a0ce-97eb850716cc","Type":"ContainerStarted","Data":"8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba"} Oct 30 21:45:51.450780 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.450738 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bvk6g" event={"ID":"191e72fd-de5a-4dea-9592-c75367da8608","Type":"ContainerStarted","Data":"a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2"} Oct 30 21:45:51.453384 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.453359 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t8l67" event={"ID":"285863f3-fa2b-4d9a-80b2-7421899e97c4","Type":"ContainerStarted","Data":"c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31"} Oct 30 21:45:51.456645 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.456619 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dql2n" event={"ID":"729a10ce-45d9-4094-8f2b-c5af2b19d159","Type":"ContainerStarted","Data":"5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487"} Oct 30 21:45:51.459252 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.459225 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqkkp" event={"ID":"cb385ed2-d5f0-4025-b614-1db5e6f0dd68","Type":"ContainerStarted","Data":"7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9"} Oct 30 21:45:51.460930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.460904 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" event={"ID":"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a","Type":"ContainerStarted","Data":"54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b"} Oct 30 21:45:51.463116 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.463092 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" event={"ID":"2e9328b7-cd5f-4a63-8895-be053051f188","Type":"ContainerStarted","Data":"6435f0bb8955515bf297cd59fc4ecb0890968609603be7fd0cf28b9c1e5a77f5"} Oct 30 21:45:51.681317 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.681199 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:50 +0000 UTC" deadline="2028-08-01 14:46:50.820437792 +0000 UTC" Oct 30 21:45:51.681317 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:51.681268 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15377h0m59.139174577s" Oct 30 21:45:53.242428 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:53.242391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:53.243054 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:53.242456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:53.243054 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.242562 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:53.243054 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.242608 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs podName:35cd7059-12c6-4c41-99ef-86eea5a5873e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:57.24259467 +0000 UTC m=+9.277163205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs") pod "network-metrics-daemon-7lq8c" (UID: "35cd7059-12c6-4c41-99ef-86eea5a5873e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:53.243054 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.242662 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:53.243054 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.242685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:57.242678169 +0000 UTC m=+9.277246703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:53.343236 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:53.343201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:53.343427 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.343406 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:53.343500 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.343435 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:53.343500 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.343449 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxksq for pod openshift-network-diagnostics/network-check-target-zbzd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:53.343604 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.343514 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq podName:90fc0a23-c03b-40f3-83ac-61ba7b26bc5e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:57.343493956 +0000 UTC m=+9.378062493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxksq" (UniqueName: "kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq") pod "network-check-target-zbzd4" (UID: "90fc0a23-c03b-40f3-83ac-61ba7b26bc5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:53.438194 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:53.438157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:53.438370 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.438304 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:45:53.438448 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:53.438391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:53.438510 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.438464 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:45:53.438578 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:53.438525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:53.438634 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:53.438611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:45:55.438315 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.438251 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:55.438808 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:55.438390 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:45:55.438808 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.438472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:55.438808 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:55.438563 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:45:55.438808 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.438613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:55.438808 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:55.438684 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:45:55.482533 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.482459 2578 generic.go:356] "Generic (PLEG): container finished" podID="f4bae0c6b7f3f9e1322bca7f9070d553" containerID="17771905f9885883d90f69d83a3f0a9e3bd4f5d0e61bd7ebb6e88de6c916a550" exitCode=0 Oct 30 21:45:55.482694 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.482548 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"f4bae0c6b7f3f9e1322bca7f9070d553","Type":"ContainerDied","Data":"17771905f9885883d90f69d83a3f0a9e3bd4f5d0e61bd7ebb6e88de6c916a550"} Oct 30 21:45:55.490152 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.490126 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"4385e5f7b5de508595fd215a6736486a","Type":"ContainerStarted","Data":"8fc10c207de2af15c1ac10e1b1c1d1881f18ff8b476005ec6f2ed0ffa163353b"} Oct 30 21:45:55.497565 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.497534 2578 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="1bddcdf233b296c0f426397bd672ef19bf9e36b9fc8b08e8bfea174a8ed4256d" exitCode=0 Oct 30 21:45:55.497672 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:55.497589 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerDied","Data":"1bddcdf233b296c0f426397bd672ef19bf9e36b9fc8b08e8bfea174a8ed4256d"} Oct 30 21:45:56.509902 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:56.509459 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"f4bae0c6b7f3f9e1322bca7f9070d553","Type":"ContainerStarted","Data":"b45955a0a68b04a68b82533c8153db49d015910665e6ed077cd7c23da30e777c"} Oct 30 21:45:56.514188 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:56.513911 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dql2n" event={"ID":"729a10ce-45d9-4094-8f2b-c5af2b19d159","Type":"ContainerStarted","Data":"5e200d7423b8fb4b9dfe0580a3de83b7a3b76f423281c113bed4cda9a9cc1e8f"} Oct 30 21:45:56.637484 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:56.637134 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" podStartSLOduration=7.637115625 podStartE2EDuration="7.637115625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:55.895559341 +0000 UTC m=+7.930127899" watchObservedRunningTime="2026-10-30 21:45:56.637115625 +0000 UTC m=+8.671684184" Oct 30 21:45:56.776512 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:56.775940 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" podStartSLOduration=7.775917493 podStartE2EDuration="7.775917493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:56.637417481 +0000 UTC m=+8.671986017" watchObservedRunningTime="2026-10-30 21:45:56.775917493 +0000 UTC m=+8.810486053" Oct 30 21:45:57.279956 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.279574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:57.279956 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.279633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:57.279956 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.279781 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:57.279956 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.279840 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs podName:35cd7059-12c6-4c41-99ef-86eea5a5873e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:05.279822717 +0000 UTC m=+17.314391266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs") pod "network-metrics-daemon-7lq8c" (UID: "35cd7059-12c6-4c41-99ef-86eea5a5873e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:45:57.279956 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.279904 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:57.279956 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.279934 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:05.279925434 +0000 UTC m=+17.314493984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:45:57.380757 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.380363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:57.380757 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.380588 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:45:57.380757 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.380608 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:45:57.380757 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.380619 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxksq for pod openshift-network-diagnostics/network-check-target-zbzd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:57.380757 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.380662 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq podName:90fc0a23-c03b-40f3-83ac-61ba7b26bc5e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:05.380649493 +0000 UTC m=+17.415218042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxksq" (UniqueName: "kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq") pod "network-check-target-zbzd4" (UID: "90fc0a23-c03b-40f3-83ac-61ba7b26bc5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:45:57.438657 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.438590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:57.438822 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.438728 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:45:57.438822 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.438603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:57.438822 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.438782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:57.439023 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.438858 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:45:57.439023 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:57.438936 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:45:57.527066 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.526479 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t8l67" event={"ID":"285863f3-fa2b-4d9a-80b2-7421899e97c4","Type":"ContainerStarted","Data":"ff321a06c44c19abdcca5d5510560f1f535b372691bbb8c8ec868ad4b64dd8ae"} Oct 30 21:45:57.656021 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:57.655903 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dql2n" podStartSLOduration=3.540431977 podStartE2EDuration="8.655879872s" totalImagesPullingTime="5.115447895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:56.776354385 +0000 UTC m=+8.810922954" watchObservedRunningTime="2026-10-30 21:45:57.655879872 +0000 UTC m=+9.690448423" Oct 30 21:45:58.530568 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:58.530530 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" event={"ID":"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a","Type":"ContainerStarted","Data":"8d54822a57aca5d08f968149515727fa4e8fb6d25d04cdade1b1f958ff941bb4"} Oct 30 21:45:58.554764 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:58.554598 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:58.555767 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:58.555623 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:45:58.685805 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:58.684930 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t8l67" podStartSLOduration=3.366354513 podStartE2EDuration="9.684910125s" totalImagesPullingTime="6.318555612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:57.656708056 +0000 UTC m=+9.691276627" watchObservedRunningTime="2026-10-30 21:45:58.684910125 +0000 UTC m=+10.719478689" Oct 30 21:45:59.437660 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:59.437625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:45:59.437842 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:59.437760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:45:59.437842 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:59.437836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:45:59.437972 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:59.437905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:45:59.437972 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:45:59.437968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:45:59.438112 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:45:59.438066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:00.180612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.180536 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:46:00.377516 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.377411 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:46:00.180567501Z","UUID":"218f8b9d-a055-47f1-95d0-9fe0b34a1e1a","Handler":null,"Name":"","Endpoint":""} Oct 30 21:46:00.379680 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.379436 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:46:00.379680 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.379465 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:46:00.535368 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.535329 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bvk6g" event={"ID":"191e72fd-de5a-4dea-9592-c75367da8608","Type":"ContainerStarted","Data":"ba76d85e31928449d4cf59ed4461356b5c7bca7fafc013896b078e742f2905b6"} Oct 30 21:46:00.537627 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.537602 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" event={"ID":"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a","Type":"ContainerStarted","Data":"999031da70c0921335761e54e4869e37843f0c4e2dd1f87b569651235422bba3"} Oct 30 21:46:00.539591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.539563 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92scn" event={"ID":"8639240c-cb7f-45b0-a0ce-97eb850716cc","Type":"ContainerStarted","Data":"6aebf3ace3fe610e29ee60ef51ab12be444c57deb5564b8ff7ff27de2651ee84"} Oct 30 21:46:00.778999 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:00.778488 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bvk6g" podStartSLOduration=3.020593864 podStartE2EDuration="11.7784698s" totalImagesPullingTime="8.757875936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.656757518 +0000 UTC m=+12.691326073" watchObservedRunningTime="2026-10-30 21:46:00.7784698 +0000 UTC m=+12.813038356" Oct 30 21:46:01.438656 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:01.438606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:01.439128 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:01.438723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:01.439128 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:01.438776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:01.439128 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:01.438878 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:01.439128 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:01.438934 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:01.439128 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:01.439014 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:01.544783 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:01.544749 2578 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="d1f54133129018ee32dcf969c732c8c7ed8b0bb5b3c4da09a49a7ad0709c5425" exitCode=0 Oct 30 21:46:01.544966 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:01.544820 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerDied","Data":"d1f54133129018ee32dcf969c732c8c7ed8b0bb5b3c4da09a49a7ad0709c5425"} Oct 30 21:46:01.671735 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:01.671684 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-92scn" podStartSLOduration=3.2896820079999998 podStartE2EDuration="12.671665966s" totalImagesPullingTime="9.381983958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.778196139 +0000 UTC m=+12.812764695" watchObservedRunningTime="2026-10-30 21:46:01.671665966 +0000 UTC m=+13.706234520" Oct 30 21:46:02.548161 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:02.548124 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqkkp" event={"ID":"cb385ed2-d5f0-4025-b614-1db5e6f0dd68","Type":"ContainerStarted","Data":"ccc37d70f214a23b190053edb182d9a7cc74df9f6433849376dfdbd2d184a6cb"} Oct 30 21:46:02.550779 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:02.550654 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" event={"ID":"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a","Type":"ContainerStarted","Data":"31e36f878fa6006438d85ee19699f7ebd1c06845d41a47bd43cb477e717f5ad1"} Oct 30 21:46:02.552309 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:02.552278 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" event={"ID":"2e9328b7-cd5f-4a63-8895-be053051f188","Type":"ContainerStarted","Data":"0fda8f567d7e78fc62e754fe33ed0dcfc4f6933bec80994605624c74c213598b"} Oct 30 21:46:02.669400 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:02.669349 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gqkkp" podStartSLOduration=4.382457613 podStartE2EDuration="13.669332524s" totalImagesPullingTime="9.286874911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:02.669199379 +0000 UTC m=+14.703767933" watchObservedRunningTime="2026-10-30 21:46:02.669332524 +0000 UTC m=+14.703901080" Oct 30 21:46:02.790359 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:02.790301 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" podStartSLOduration=2.416261168 podStartE2EDuration="13.790283463s" totalImagesPullingTime="11.374022295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:02.789016274 +0000 UTC m=+14.823584828" watchObservedRunningTime="2026-10-30 21:46:02.790283463 +0000 UTC m=+14.824852019" Oct 30 21:46:02.911244 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:02.911188 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" podStartSLOduration=3.341624216 podStartE2EDuration="13.911167214s" totalImagesPullingTime="10.569542998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:02.910714805 +0000 UTC m=+14.945283371" watchObservedRunningTime="2026-10-30 21:46:02.911167214 +0000 UTC m=+14.945735770" Oct 30 21:46:03.438005 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:03.437970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:03.438161 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:03.438047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:03.438161 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:03.438072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:03.438255 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:03.438189 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:03.438289 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:03.438252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:03.438289 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:03.438263 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:03.556547 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:03.556511 2578 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="a56ea0e17c21ce2cd87a68d9619f6960409e2fb1fe1b552d59e9e5324e489eac" exitCode=0 Oct 30 21:46:03.557115 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:03.556607 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerDied","Data":"a56ea0e17c21ce2cd87a68d9619f6960409e2fb1fe1b552d59e9e5324e489eac"} Oct 30 21:46:05.345384 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.345349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:05.345939 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.345401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:05.345939 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.345492 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:46:05.345939 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.345494 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:46:05.345939 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.345537 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs podName:35cd7059-12c6-4c41-99ef-86eea5a5873e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.345523289 +0000 UTC m=+33.380091822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs") pod "network-metrics-daemon-7lq8c" (UID: "35cd7059-12c6-4c41-99ef-86eea5a5873e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:46:05.345939 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.345553 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.345545508 +0000 UTC m=+33.380114043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:46:05.437536 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.437506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:05.437687 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.437510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:05.437687 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.437634 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:05.437798 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.437506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:05.437844 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.437687 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:05.437844 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.437797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:05.445817 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.445790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:05.445968 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.445938 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:46:05.446042 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.445975 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:46:05.446042 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.445988 2578 projected.go:196] Error preparing data for projected volume kube-api-access-hxksq for pod openshift-network-diagnostics/network-check-target-zbzd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:46:05.446042 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:05.446036 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq podName:90fc0a23-c03b-40f3-83ac-61ba7b26bc5e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:21.446019018 +0000 UTC m=+33.480587568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxksq" (UniqueName: "kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq") pod "network-check-target-zbzd4" (UID: "90fc0a23-c03b-40f3-83ac-61ba7b26bc5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:46:05.564617 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.564575 2578 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="ab26072c248c6c77497adc6eb63ff7d7a9f748d54a4236c565a3463ba06d8d37" exitCode=0 Oct 30 21:46:05.564744 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:05.564640 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerDied","Data":"ab26072c248c6c77497adc6eb63ff7d7a9f748d54a4236c565a3463ba06d8d37"} Oct 30 21:46:06.569048 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.568860 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:46:06.569460 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569395 2578 generic.go:356] "Generic (PLEG): container finished" podID="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" containerID="1962f31cdcbb3fc9bc6ec7676c633c70bbea935bce89a263ea07d86062101529" exitCode=1 Oct 30 21:46:06.569460 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569438 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"4bb3bbc90c435acd480824513fe89a99e46b8999eac9795bbb4699ef2a848ad2"} Oct 30 21:46:06.569541 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569504 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"932013e4d203f80763a0456d1e832d761accbedbaefbca1e6d56c4c03a0ee126"} Oct 30 21:46:06.569541 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"759cc912823e2072ed70056afcb479e60175edf03ba15cc2bacde8db8901c600"} Oct 30 21:46:06.569598 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569562 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"c0ee9fd0fb3ed62243c690bc31ec5a29d261fd5637bff4a9690bbac7c3819533"} Oct 30 21:46:06.569598 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569589 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerDied","Data":"1962f31cdcbb3fc9bc6ec7676c633c70bbea935bce89a263ea07d86062101529"} Oct 30 21:46:06.569651 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:06.569616 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"acb0301ce7a3df7cb9518526e7e315409d2999ed3288b36c1edec6b4fe92a61a"} Oct 30 21:46:07.438569 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:07.438359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:07.438727 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:07.438420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:07.438727 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:07.438662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:07.438727 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:07.438426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:07.438891 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:07.438730 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:07.438891 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:07.438818 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:08.575922 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:08.575848 2578 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="f7d7a25b4a897a7aae3f56175e5c3ce9b37349bce69285502cafffd63f143420" exitCode=0 Oct 30 21:46:08.576415 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:08.575921 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerDied","Data":"f7d7a25b4a897a7aae3f56175e5c3ce9b37349bce69285502cafffd63f143420"} Oct 30 21:46:08.579028 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:08.579010 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:46:08.579424 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:08.579391 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"9db4d35657c9de7d2704013ff2bdffb6ff9eb5ec25bd87ec0d5a1423d8c5eb33"} Oct 30 21:46:09.438438 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:09.438410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:09.438578 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:09.438411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:09.438578 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:09.438517 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:09.438578 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:09.438539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:09.438693 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:09.438614 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:09.438693 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:09.438681 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:09.585500 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:09.585475 2578 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="b9e6d102ee9925d5ed302db57a78f19b911c25dbcf91795595bc4bdd31b99892" exitCode=0 Oct 30 21:46:09.585850 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:09.585521 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerDied","Data":"b9e6d102ee9925d5ed302db57a78f19b911c25dbcf91795595bc4bdd31b99892"} Oct 30 21:46:10.592518 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.592493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:46:10.593403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.592920 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"6a84b07be76c38fc46e604918295960c38c4733135479d864f92b41c37872fb4"} Oct 30 21:46:10.593403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.593244 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:46:10.593403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.593307 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:46:10.593403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.593310 2578 scope.go:122] "RemoveContainer" containerID="1962f31cdcbb3fc9bc6ec7676c633c70bbea935bce89a263ea07d86062101529" Oct 30 21:46:10.593403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.593340 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:46:10.597163 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.597137 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" event={"ID":"248fb46a-f3b1-4d89-9d88-0191280d983d","Type":"ContainerStarted","Data":"9721771a1a6c5d86b304dde51d76a884d5b13e29d953d816838e895aaf3ae1ae"} Oct 30 21:46:10.611625 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.611606 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:46:10.613595 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:10.613578 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:46:11.130276 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.130103 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8xjbc" podStartSLOduration=8.541431296 podStartE2EDuration="22.130091478s" totalImagesPullingTime="13.588660182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:11.130000533 +0000 UTC m=+23.164569086" watchObservedRunningTime="2026-10-30 21:46:11.130091478 +0000 UTC m=+23.164660033" Oct 30 21:46:11.438397 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.438369 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:11.438615 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.438510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:11.438615 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:11.438530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:11.438615 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:11.438574 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:11.438783 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.438623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:11.438783 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:11.438752 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:11.602993 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.602975 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:46:11.603402 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.603382 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" event={"ID":"18cf0a1a-efb8-4aef-ac2a-16e294018ccf","Type":"ContainerStarted","Data":"12c83e9959527735d10d964bf045a97f4ad70d236a4be59b0d8bc4b75c06ed65"} Oct 30 21:46:11.788788 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:11.788696 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" podStartSLOduration=6.959614094 podStartE2EDuration="22.788681286s" totalImagesPullingTime="15.829067192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:11.788187836 +0000 UTC m=+23.822756403" watchObservedRunningTime="2026-10-30 21:46:11.788681286 +0000 UTC m=+23.823249842" Oct 30 21:46:12.833705 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:12.833673 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:46:12.834230 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:12.833865 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:46:12.834230 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:12.834168 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:46:13.108336 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:13.108274 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zbzd4"] Oct 30 21:46:13.108459 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:13.108379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:13.108496 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:13.108464 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:13.110612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:13.110590 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-687l6"] Oct 30 21:46:13.110701 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:13.110689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:13.110806 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:13.110787 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:13.111264 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:13.111246 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7lq8c"] Oct 30 21:46:13.111339 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:13.111329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:13.111414 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:13.111400 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:14.437918 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:14.437887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:14.438383 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:14.437887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:14.438383 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:14.438065 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:14.438383 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:14.438081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:15.437876 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:15.437856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:15.438013 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:15.437985 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:16.438323 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:16.438288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:16.438688 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:16.438418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:16.438688 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:16.438465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:16.438688 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:16.438575 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:17.438396 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:17.438361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:17.438770 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:17.438466 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:18.439304 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:18.439265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:18.439746 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:18.439347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:18.439746 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:18.439398 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbzd4" podUID="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" Oct 30 21:46:18.439746 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:18.439393 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lq8c" podUID="35cd7059-12c6-4c41-99ef-86eea5a5873e" Oct 30 21:46:19.368479 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:19.368335 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-92scn_8639240c-cb7f-45b0-a0ce-97eb850716cc/dns-node-resolver/0.log" Oct 30 21:46:19.438544 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:19.438519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:19.438634 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:19.438617 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-687l6" podUID="1bdaec03-c2f8-446e-9376-bed9a55677c9" Oct 30 21:46:19.747510 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:19.747491 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeReady" Oct 30 21:46:19.747829 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:19.747593 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:46:20.341552 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.341529 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t8l67_285863f3-fa2b-4d9a-80b2-7421899e97c4/node-ca/0.log" Oct 30 21:46:20.438365 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.438344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:20.438612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.438577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:20.492842 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.492819 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:46:20.492916 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.492843 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:46:20.493406 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.493393 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:46:20.524119 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.524099 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\"" Oct 30 21:46:20.525301 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:20.525286 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\"" Oct 30 21:46:21.359125 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.359093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:21.359673 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.359136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:21.359673 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:21.359264 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:46:21.359673 ip-10-0-133-91 kubenswrapper[2578]: E1030 21:46:21.359335 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret podName:1bdaec03-c2f8-446e-9376-bed9a55677c9 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:53.359319424 +0000 UTC m=+65.393887956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret") pod "global-pull-secret-syncer-687l6" (UID: "1bdaec03-c2f8-446e-9376-bed9a55677c9") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:46:21.363632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.363609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:21.407794 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.407759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:46:21.437678 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.437649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:21.459847 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.459825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:21.462611 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.462580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:21.496215 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.496195 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\"" Oct 30 21:46:21.511031 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.511010 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:46:21.673985 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.673913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:21.789741 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:21.789716 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cd7059_12c6_4c41_99ef_86eea5a5873e.slice/crio-b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1 WatchSource:0}: Error finding container b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1: Status 404 returned error can't find the container with id b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1 Oct 30 21:46:21.790391 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:21.790358 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7lq8c"] Oct 30 21:46:22.063613 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:22.063577 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fc0a23_c03b_40f3_83ac_61ba7b26bc5e.slice/crio-bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8 WatchSource:0}: Error finding container bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8: Status 404 returned error can't find the container with id bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8 Oct 30 21:46:22.064461 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:22.064430 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zbzd4"] Oct 30 21:46:22.626019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:22.625989 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zbzd4" event={"ID":"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e","Type":"ContainerStarted","Data":"bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8"} Oct 30 21:46:22.627185 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:22.627162 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lq8c" event={"ID":"35cd7059-12c6-4c41-99ef-86eea5a5873e","Type":"ContainerStarted","Data":"b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1"} Oct 30 21:46:23.632165 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:23.632130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lq8c" event={"ID":"35cd7059-12c6-4c41-99ef-86eea5a5873e","Type":"ContainerStarted","Data":"d5eb3f424afc07aa557dddf593530a9d32867567a379d164b05565f1ea7ffd4e"} Oct 30 21:46:23.632578 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:23.632184 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lq8c" event={"ID":"35cd7059-12c6-4c41-99ef-86eea5a5873e","Type":"ContainerStarted","Data":"337769d2bfaaba851561099b9c7425c618b534e14203f5e56cd8a5453bb522ab"} Oct 30 21:46:23.763504 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:23.763457 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7lq8c" podStartSLOduration=33.152959225 podStartE2EDuration="34.763443498s" totalImagesPullingTime="1.610484273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:23.762429586 +0000 UTC m=+35.796998140" watchObservedRunningTime="2026-10-30 21:46:23.763443498 +0000 UTC m=+35.798012049" Oct 30 21:46:25.638435 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:25.638398 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zbzd4" event={"ID":"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e","Type":"ContainerStarted","Data":"1f0af5139c5ba57532f2b54caebefdb6ea6a74efb6aaea04465b571a5ec91406"} Oct 30 21:46:25.638830 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:25.638550 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:25.758098 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:25.758054 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zbzd4" podStartSLOduration=33.40279821 podStartE2EDuration="36.758042053s" totalImagesPullingTime="3.355243843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:25.757363519 +0000 UTC m=+37.791932093" watchObservedRunningTime="2026-10-30 21:46:25.758042053 +0000 UTC m=+37.792610608" Oct 30 21:46:39.752470 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.752437 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cjx5v"] Oct 30 21:46:39.793499 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.793471 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-phd9k"] Oct 30 21:46:39.793652 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.793632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.809075 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.809055 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-544774fdb7-dzpt4"] Oct 30 21:46:39.809199 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.809176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.823301 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.823282 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-jl9q2"] Oct 30 21:46:39.823393 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.823362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.844462 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.844441 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4"] Oct 30 21:46:39.844571 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.844555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:39.851030 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.851009 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:46:39.853142 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.853124 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:46:39.853142 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.853124 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hs26f\"" Oct 30 21:46:39.858596 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.858579 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4f8cw"] Oct 30 21:46:39.858714 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.858699 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:39.872402 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.872491 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872410 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80e5d657-00d6-433e-b9bd-0ca172deb64d-metrics-tls\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.872491 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-data-volume\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.872491 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-crio-socket\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.872491 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5bq\" (UniqueName: \"kubernetes.io/projected/80e5d657-00d6-433e-b9bd-0ca172deb64d-kube-api-access-gg5bq\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.872629 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.872629 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prh52\" (UniqueName: \"kubernetes.io/projected/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-api-access-prh52\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.872691 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80e5d657-00d6-433e-b9bd-0ca172deb64d-config-volume\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.872691 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.872682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e5d657-00d6-433e-b9bd-0ca172deb64d-tmp-dir\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.878919 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.878901 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c78b64cb4-4q2c8"] Oct 30 21:46:39.879039 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.879024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:39.887286 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.887269 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:46:39.887371 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.887299 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:46:39.887371 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.887301 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:46:39.887523 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.887508 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:46:39.891115 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.891094 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:46:39.891195 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.891132 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:46:39.903362 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.903301 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cjx5v"] Oct 30 21:46:39.903362 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.903342 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phd9k"] Oct 30 21:46:39.903505 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.903456 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:46:39.903737 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.903721 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:46:39.904460 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.904440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.908417 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.908394 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-544774fdb7-dzpt4"] Oct 30 21:46:39.908610 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.908596 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-jl9q2"] Oct 30 21:46:39.908727 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.908716 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4"] Oct 30 21:46:39.910366 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.910348 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4f8cw"] Oct 30 21:46:39.932141 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.932125 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:46:39.932247 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.932233 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:46:39.952211 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.952177 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cpxgn\"" Oct 30 21:46:39.952211 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.952195 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:46:39.952428 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.952415 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:46:39.952575 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.952561 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:46:39.952625 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.952594 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:46:39.952658 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.952648 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-879p5\"" Oct 30 21:46:39.954300 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.954288 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:46:39.954300 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.954294 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hlprr\"" Oct 30 21:46:39.954424 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.954411 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qq6nx\"" Oct 30 21:46:39.954459 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.954444 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pz9m9\"" Oct 30 21:46:39.973613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973592 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c78b64cb4-4q2c8"] Oct 30 21:46:39.973689 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.973689 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcwv\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-kube-api-access-txcwv\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.973780 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80e5d657-00d6-433e-b9bd-0ca172deb64d-config-volume\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.973780 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e5d657-00d6-433e-b9bd-0ca172deb64d-tmp-dir\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.973851 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vl64\" (UniqueName: \"kubernetes.io/projected/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-kube-api-access-4vl64\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:39.973946 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-image-registry-private-configuration\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.973975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-tls\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.974100 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80e5d657-00d6-433e-b9bd-0ca172deb64d-metrics-tls\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.974100 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e5d657-00d6-433e-b9bd-0ca172deb64d-tmp-dir\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.974199 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-data-volume\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.974199 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5bq\" (UniqueName: \"kubernetes.io/projected/80e5d657-00d6-433e-b9bd-0ca172deb64d-kube-api-access-gg5bq\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.974285 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.974285 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-certificates\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974285 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5619b33-0539-4471-ac48-e5f51da5ab38-ca-trust-extracted\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974423 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prh52\" (UniqueName: \"kubernetes.io/projected/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-api-access-prh52\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.974423 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.974423 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-cert\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:39.974423 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e46f2-7b2a-4518-861c-fb94540b0231-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tgqt4\" (UID: \"6f4e46f2-7b2a-4518-861c-fb94540b0231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-trusted-ca\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-installation-pull-secrets\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-crio-socket\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.974613 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdxt\" (UniqueName: \"kubernetes.io/projected/a8dbcc04-78d4-4ac0-8697-deec99270f95-kube-api-access-9hdxt\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:39.974927 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:39.974927 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-bound-sa-token\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:39.974927 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8dbcc04-78d4-4ac0-8697-deec99270f95-tmp\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:39.974927 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-data-volume\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.975150 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.974369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80e5d657-00d6-433e-b9bd-0ca172deb64d-config-volume\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.975150 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.975052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-crio-socket\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.975671 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.975645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:39.976723 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.976702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80e5d657-00d6-433e-b9bd-0ca172deb64d-metrics-tls\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:39.976796 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:39.976712 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:40.003455 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.003401 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:46:40.039113 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.039089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5bq\" (UniqueName: \"kubernetes.io/projected/80e5d657-00d6-433e-b9bd-0ca172deb64d-kube-api-access-gg5bq\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:40.040050 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.040031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prh52\" (UniqueName: \"kubernetes.io/projected/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-api-access-prh52\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:40.054968 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.054930 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:46:40.055193 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.055175 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:46:40.055257 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.055197 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:46:40.055257 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.055175 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g8m85\"" Oct 30 21:46:40.075559 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e46f2-7b2a-4518-861c-fb94540b0231-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tgqt4\" (UID: \"6f4e46f2-7b2a-4518-861c-fb94540b0231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:40.075650 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.075650 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-trusted-ca\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.075650 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.075650 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-installation-pull-secrets\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdxt\" (UniqueName: \"kubernetes.io/projected/a8dbcc04-78d4-4ac0-8697-deec99270f95-kube-api-access-9hdxt\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-bound-sa-token\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8dbcc04-78d4-4ac0-8697-deec99270f95-tmp\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:40.075843 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.076192 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcwv\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-kube-api-access-txcwv\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.076192 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl64\" (UniqueName: \"kubernetes.io/projected/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-kube-api-access-4vl64\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:40.076192 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.075912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-image-registry-private-configuration\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.076192 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.076049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.076396 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.076339 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.076794 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.076771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.077038 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.077013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-trusted-ca\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.077195 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.077049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.077373 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.077244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-tls\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.077592 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.077545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-certificates\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5619b33-0539-4471-ac48-e5f51da5ab38-ca-trust-extracted\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-cert\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-certificates\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.077647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8dbcc04-78d4-4ac0-8697-deec99270f95-tmp\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078517 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.078738 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5619b33-0539-4471-ac48-e5f51da5ab38-ca-trust-extracted\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.079099 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.078883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-image-registry-private-configuration\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.079099 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.079068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-installation-pull-secrets\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.079171 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.079100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.079768 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.079743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e46f2-7b2a-4518-861c-fb94540b0231-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tgqt4\" (UID: \"6f4e46f2-7b2a-4518-861c-fb94540b0231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:40.080004 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.079985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-tls\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.080465 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.080449 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-cert\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:40.128207 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.128179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:40.138588 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.138567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdxt\" (UniqueName: \"kubernetes.io/projected/a8dbcc04-78d4-4ac0-8697-deec99270f95-kube-api-access-9hdxt\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:40.138999 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.138982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-bound-sa-token\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.139250 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.139227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcwv\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-kube-api-access-txcwv\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.139333 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.139231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl64\" (UniqueName: \"kubernetes.io/projected/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-kube-api-access-4vl64\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:40.141160 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.141134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.173612 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.173589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:46:40.189911 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.189888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:40.212638 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.212379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:46:40.231144 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.230911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:40.232873 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.232718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:46:40.236607 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.234514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:40.505773 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.505718 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e5d657_00d6_433e_b9bd_0ca172deb64d.slice/crio-9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50 WatchSource:0}: Error finding container 9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50: Status 404 returned error can't find the container with id 9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50 Oct 30 21:46:40.506720 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.506699 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cjx5v"] Oct 30 21:46:40.541275 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.541250 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd3fa78_0fa0_41de_b566_bc0cea25b5b4.slice/crio-adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85 WatchSource:0}: Error finding container adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85: Status 404 returned error can't find the container with id adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85 Oct 30 21:46:40.542607 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.542589 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phd9k"] Oct 30 21:46:40.567382 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.567357 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-544774fdb7-dzpt4"] Oct 30 21:46:40.568171 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.568151 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb23260_4a0e_44e6_8023_124e58c32c5e.slice/crio-8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2 WatchSource:0}: Error finding container 8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2: Status 404 returned error can't find the container with id 8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2 Oct 30 21:46:40.615091 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.615062 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-jl9q2"] Oct 30 21:46:40.615853 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.615832 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dbcc04_78d4_4ac0_8697_deec99270f95.slice/crio-aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb WatchSource:0}: Error finding container aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb: Status 404 returned error can't find the container with id aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb Oct 30 21:46:40.618035 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.618012 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e46f2_7b2a_4518_861c_fb94540b0231.slice/crio-4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830 WatchSource:0}: Error finding container 4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830: Status 404 returned error can't find the container with id 4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830 Oct 30 21:46:40.618242 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.618224 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4"] Oct 30 21:46:40.635940 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.635918 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0140305c_d4d1_4ae9_877a_6c6d21b80d0d.slice/crio-df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297 WatchSource:0}: Error finding container df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297: Status 404 returned error can't find the container with id df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297 Oct 30 21:46:40.641015 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.640991 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4f8cw"] Oct 30 21:46:40.643709 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.643661 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c78b64cb4-4q2c8"] Oct 30 21:46:40.645241 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:40.645222 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5619b33_0539_4471_ac48_e5f51da5ab38.slice/crio-8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b WatchSource:0}: Error finding container 8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b: Status 404 returned error can't find the container with id 8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b Oct 30 21:46:40.673338 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.673315 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-jl9q2" event={"ID":"a8dbcc04-78d4-4ac0-8697-deec99270f95","Type":"ContainerStarted","Data":"aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb"} Oct 30 21:46:40.674886 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.674865 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phd9k" event={"ID":"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4","Type":"ContainerStarted","Data":"c1a576228f49081b735695f0888cd2bb23745634c2cc9cc0c030ef1762a0db30"} Oct 30 21:46:40.674981 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.674913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phd9k" event={"ID":"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4","Type":"ContainerStarted","Data":"adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85"} Oct 30 21:46:40.676094 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.676075 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" event={"ID":"b5619b33-0539-4471-ac48-e5f51da5ab38","Type":"ContainerStarted","Data":"8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b"} Oct 30 21:46:40.677227 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.677207 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4f8cw" event={"ID":"0140305c-d4d1-4ae9-877a-6c6d21b80d0d","Type":"ContainerStarted","Data":"df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297"} Oct 30 21:46:40.678425 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.678402 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544774fdb7-dzpt4" event={"ID":"8cb23260-4a0e-44e6-8023-124e58c32c5e","Type":"ContainerStarted","Data":"8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2"} Oct 30 21:46:40.679639 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.679616 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cjx5v" event={"ID":"80e5d657-00d6-433e-b9bd-0ca172deb64d","Type":"ContainerStarted","Data":"9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50"} Oct 30 21:46:40.681352 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:40.680899 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" event={"ID":"6f4e46f2-7b2a-4518-861c-fb94540b0231","Type":"ContainerStarted","Data":"4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830"} Oct 30 21:46:41.686720 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:41.686681 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" event={"ID":"b5619b33-0539-4471-ac48-e5f51da5ab38","Type":"ContainerStarted","Data":"303db3d2d1c061416abdbbd47611fe643ae671af638c9455545916e7b6b4ae57"} Oct 30 21:46:41.687255 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:41.686992 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:46:41.810850 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:41.810204 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" podStartSLOduration=2.8101852000000003 podStartE2EDuration="2.8101852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:41.809054638 +0000 UTC m=+53.843623197" watchObservedRunningTime="2026-10-30 21:46:41.8101852 +0000 UTC m=+53.844753772" Oct 30 21:46:42.626327 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:42.626293 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:46:42.695791 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:42.695762 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phd9k" event={"ID":"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4","Type":"ContainerStarted","Data":"6415c035569acdb78568f916e58c334caef0cce7df4197e9a8bbb7ef07e16183"} Oct 30 21:46:43.723131 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:43.723083 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" event={"ID":"6f4e46f2-7b2a-4518-861c-fb94540b0231","Type":"ContainerStarted","Data":"b10369ba4486e3e0378582151146436d7d6dfa07c83186b8349043ee9c4beb53"} Oct 30 21:46:43.723498 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:43.723235 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:43.729257 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:43.729236 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:46:43.972893 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:43.972819 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" podStartSLOduration=2.786153014 podStartE2EDuration="4.97280669s" totalImagesPullingTime="2.186653676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:43.840356683 +0000 UTC m=+55.874925238" watchObservedRunningTime="2026-10-30 21:46:43.97280669 +0000 UTC m=+56.007375245" Oct 30 21:46:44.730544 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:44.730510 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cjx5v" event={"ID":"80e5d657-00d6-433e-b9bd-0ca172deb64d","Type":"ContainerStarted","Data":"d5c6d6bcff52281f635a4fdd7c630a1314ea4fcc8693c058bb773f2da660b50c"} Oct 30 21:46:44.731031 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:44.730562 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cjx5v" event={"ID":"80e5d657-00d6-433e-b9bd-0ca172deb64d","Type":"ContainerStarted","Data":"b39a638633293b19acbefccbe14d2e2e57def77403a5817ff1a2c462f1b25639"} Oct 30 21:46:44.731031 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:44.730620 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:44.853223 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:44.853163 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-cjx5v" podStartSLOduration=2.639162175 podStartE2EDuration="5.853144492s" totalImagesPullingTime="3.213982317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:44.852424791 +0000 UTC m=+56.886993347" watchObservedRunningTime="2026-10-30 21:46:44.853144492 +0000 UTC m=+56.887713050" Oct 30 21:46:45.736701 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:45.736663 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phd9k" event={"ID":"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4","Type":"ContainerStarted","Data":"e59357dd55cf6de773980c101ff2c7c5e4f222c5c94aba91934e34bf7023bcca"} Oct 30 21:46:45.738439 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:45.738384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4f8cw" event={"ID":"0140305c-d4d1-4ae9-877a-6c6d21b80d0d","Type":"ContainerStarted","Data":"3589e2975054490d47b6b82f10a1609ae3e8c5a4afd7b79533104cfb32f3f76c"} Oct 30 21:46:45.740183 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:45.740151 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544774fdb7-dzpt4" event={"ID":"8cb23260-4a0e-44e6-8023-124e58c32c5e","Type":"ContainerStarted","Data":"23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44"} Oct 30 21:46:45.864632 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:45.864590 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-phd9k" podStartSLOduration=2.739339061 podStartE2EDuration="6.864576742s" totalImagesPullingTime="4.125237681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:45.863692887 +0000 UTC m=+57.898261444" watchObservedRunningTime="2026-10-30 21:46:45.864576742 +0000 UTC m=+57.899145297" Oct 30 21:46:45.985879 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:45.985823 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4f8cw" podStartSLOduration=2.190976914 podStartE2EDuration="6.985806605s" totalImagesPullingTime="4.794829691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:45.985584602 +0000 UTC m=+58.020153157" watchObservedRunningTime="2026-10-30 21:46:45.985806605 +0000 UTC m=+58.020375160" Oct 30 21:46:46.106829 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:46.106724 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-544774fdb7-dzpt4" podStartSLOduration=2.254096385 podStartE2EDuration="7.106713771s" totalImagesPullingTime="4.852617386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:46.106105736 +0000 UTC m=+58.140674293" watchObservedRunningTime="2026-10-30 21:46:46.106713771 +0000 UTC m=+58.141282326" Oct 30 21:46:50.190824 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:50.190786 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:50.191266 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:50.190858 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:50.196211 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:50.196186 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:50.760415 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:50.760382 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:46:52.084963 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.084930 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh"] Oct 30 21:46:52.087162 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.087145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.145931 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.145906 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh"] Oct 30 21:46:52.182560 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.182531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.182671 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.182651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkft\" (UniqueName: \"kubernetes.io/projected/ffe895c1-e716-45d5-94eb-58668c4bd1c1-kube-api-access-dqkft\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.182727 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.182693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe895c1-e716-45d5-94eb-58668c4bd1c1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.182727 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.182712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.195431 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.195413 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:46:52.195591 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.195570 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:46:52.195703 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.195677 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:46:52.195769 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.195721 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:46:52.196407 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.196371 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:46:52.196407 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.196388 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mpsf4\"" Oct 30 21:46:52.247514 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.247494 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fq2xz"] Oct 30 21:46:52.249568 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.249546 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.284505 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.283737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkft\" (UniqueName: \"kubernetes.io/projected/ffe895c1-e716-45d5-94eb-58668c4bd1c1-kube-api-access-dqkft\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.284505 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.283851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe895c1-e716-45d5-94eb-58668c4bd1c1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.284505 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.283893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.284505 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.283943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.284761 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.284622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe895c1-e716-45d5-94eb-58668c4bd1c1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.287009 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.286986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.291019 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.290996 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.315101 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.315085 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:46:52.315805 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.315789 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4gblc\"" Oct 30 21:46:52.317112 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.317092 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:46:52.347262 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.347199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkft\" (UniqueName: \"kubernetes.io/projected/ffe895c1-e716-45d5-94eb-58668c4bd1c1-kube-api-access-dqkft\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.385144 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385268 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-sys\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385337 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-wtmp\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385389 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-metrics-client-ca\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385423 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcbwm\" (UniqueName: \"kubernetes.io/projected/ab980c40-b10c-4f42-90d8-bd930becdb72-kube-api-access-vcbwm\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385517 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-tls\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385585 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-root\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385634 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-textfile\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.385677 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.385657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.400784 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.400762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:46:52.411046 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.411021 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:46:52.486394 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-root\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486513 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-textfile\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486513 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486513 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486513 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-sys\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486513 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-wtmp\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486666 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-metrics-client-ca\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486666 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcbwm\" (UniqueName: \"kubernetes.io/projected/ab980c40-b10c-4f42-90d8-bd930becdb72-kube-api-access-vcbwm\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486666 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-tls\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486977 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-root\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486977 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-wtmp\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.486977 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.486928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-sys\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.487188 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.487046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-textfile\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.487518 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.487495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.487777 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.487751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-metrics-client-ca\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.488790 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.488771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-tls\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.489099 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.489081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.550423 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.550397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcbwm\" (UniqueName: \"kubernetes.io/projected/ab980c40-b10c-4f42-90d8-bd930becdb72-kube-api-access-vcbwm\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.564215 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.564195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:46:52.574885 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:52.574810 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab980c40_b10c_4f42_90d8_bd930becdb72.slice/crio-3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754 WatchSource:0}: Error finding container 3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754: Status 404 returned error can't find the container with id 3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754 Oct 30 21:46:52.764073 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.764033 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq2xz" event={"ID":"ab980c40-b10c-4f42-90d8-bd930becdb72","Type":"ContainerStarted","Data":"3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754"} Oct 30 21:46:52.817789 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:52.817762 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe895c1_e716_45d5_94eb_58668c4bd1c1.slice/crio-3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8 WatchSource:0}: Error finding container 3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8: Status 404 returned error can't find the container with id 3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8 Oct 30 21:46:52.818671 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:52.818648 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh"] Oct 30 21:46:53.394234 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.394196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:53.447401 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.447367 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:46:53.458087 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.458060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:53.601516 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.601484 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\"" Oct 30 21:46:53.609545 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.609520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:46:53.770432 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.770393 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" event={"ID":"ffe895c1-e716-45d5-94eb-58668c4bd1c1","Type":"ContainerStarted","Data":"bde456b44d8166f569ce58638394236dc0cfd8af83e40e229fab7181013babcd"} Oct 30 21:46:53.770593 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.770476 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" event={"ID":"ffe895c1-e716-45d5-94eb-58668c4bd1c1","Type":"ContainerStarted","Data":"e469dac708c14a7c5a2541ab422f21f8badf7758a4ebc8d74af2373392abc2ee"} Oct 30 21:46:53.770593 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.770507 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" event={"ID":"ffe895c1-e716-45d5-94eb-58668c4bd1c1","Type":"ContainerStarted","Data":"3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8"} Oct 30 21:46:53.995804 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:53.995775 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdaec03_c2f8_446e_9376_bed9a55677c9.slice/crio-ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d WatchSource:0}: Error finding container ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d: Status 404 returned error can't find the container with id ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d Oct 30 21:46:53.996884 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:53.996857 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-687l6"] Oct 30 21:46:54.775488 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.775460 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f9b796bff-x4mrn"] Oct 30 21:46:54.778863 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.778837 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" event={"ID":"ffe895c1-e716-45d5-94eb-58668c4bd1c1","Type":"ContainerStarted","Data":"8c33b2a35f584d4a46e06b8e0840e6d402214ee5cbe20c8f0062b600f8cdadbf"} Oct 30 21:46:54.778995 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.778983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.779730 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.779695 2578 generic.go:356] "Generic (PLEG): container finished" podID="ab980c40-b10c-4f42-90d8-bd930becdb72" containerID="c6828a656afe137548eb5bc42fd653c14a918df3ef8e15be60b45a6cfeb98013" exitCode=0 Oct 30 21:46:54.779824 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.779792 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq2xz" event={"ID":"ab980c40-b10c-4f42-90d8-bd930becdb72","Type":"ContainerDied","Data":"c6828a656afe137548eb5bc42fd653c14a918df3ef8e15be60b45a6cfeb98013"} Oct 30 21:46:54.782794 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.782764 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-687l6" event={"ID":"1bdaec03-c2f8-446e-9376-bed9a55677c9","Type":"ContainerStarted","Data":"ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d"} Oct 30 21:46:54.833820 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.833793 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:46:54.833924 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.833823 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:46:54.833996 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.833965 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:46:54.833996 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.833982 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a50jmvgl924dq\"" Oct 30 21:46:54.881732 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.881703 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:46:54.881851 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.881722 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:46:54.881851 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.881745 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vvlch\"" Oct 30 21:46:54.883258 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.883233 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f9b796bff-x4mrn"] Oct 30 21:46:54.900200 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.900154 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" podStartSLOduration=1.272419315 podStartE2EDuration="2.90014161s" totalImagesPullingTime="1.627722295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.899282407 +0000 UTC m=+66.933850961" watchObservedRunningTime="2026-10-30 21:46:54.90014161 +0000 UTC m=+66.934710164" Oct 30 21:46:54.906844 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.906822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.906930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.906884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.906930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.906914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.907082 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.907003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfccd4-cbd5-4319-be94-3da3df1d8566-metrics-client-ca\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.907082 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.907020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ebfccd4-cbd5-4319-be94-3da3df1d8566-tmp\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.907082 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.907060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.907298 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.907115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.907298 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.907135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-grpc-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:54.907298 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:54.907178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6bl\" (UniqueName: \"kubernetes.io/projected/3ebfccd4-cbd5-4319-be94-3da3df1d8566-kube-api-access-7k6bl\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008533 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008698 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-grpc-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008698 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6bl\" (UniqueName: \"kubernetes.io/projected/3ebfccd4-cbd5-4319-be94-3da3df1d8566-kube-api-access-7k6bl\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008698 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008698 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008698 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008941 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfccd4-cbd5-4319-be94-3da3df1d8566-metrics-client-ca\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008941 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ebfccd4-cbd5-4319-be94-3da3df1d8566-tmp\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.008941 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.008788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.010171 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.010143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ebfccd4-cbd5-4319-be94-3da3df1d8566-tmp\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.010351 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.010323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfccd4-cbd5-4319-be94-3da3df1d8566-metrics-client-ca\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.011982 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.011921 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.012189 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.012171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.012442 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.012416 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.012863 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.012837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.012991 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.012930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-grpc-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.014556 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.014534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.073321 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.073278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6bl\" (UniqueName: \"kubernetes.io/projected/3ebfccd4-cbd5-4319-be94-3da3df1d8566-kube-api-access-7k6bl\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.097217 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.097156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:46:55.537724 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:55.537689 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebfccd4_cbd5_4319_be94_3da3df1d8566.slice/crio-c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298 WatchSource:0}: Error finding container c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298: Status 404 returned error can't find the container with id c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298 Oct 30 21:46:55.538182 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.538156 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f9b796bff-x4mrn"] Oct 30 21:46:55.745933 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.745903 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cjx5v" Oct 30 21:46:55.788979 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.788879 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298"} Oct 30 21:46:55.793466 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.793436 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq2xz" event={"ID":"ab980c40-b10c-4f42-90d8-bd930becdb72","Type":"ContainerStarted","Data":"b1353aa69e5f3a80cb97a221f820300832452ebae947d5a766ba967f4ce2a107"} Oct 30 21:46:55.793589 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.793483 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fq2xz" event={"ID":"ab980c40-b10c-4f42-90d8-bd930becdb72","Type":"ContainerStarted","Data":"337ea854dbfb0abf838571e87b46e0fec2cd46190f65f7a4be8cff2e5486417f"} Oct 30 21:46:55.989611 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:55.989557 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fq2xz" podStartSLOduration=2.384592674 podStartE2EDuration="3.989543604s" totalImagesPullingTime="1.512422032s" totalInitContainerRuntime="92.528898ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:55.987899068 +0000 UTC m=+68.022467626" watchObservedRunningTime="2026-10-30 21:46:55.989543604 +0000 UTC m=+68.024112151" Oct 30 21:46:56.643740 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:56.643708 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:46:57.121083 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.119480 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-664c5b7ff9-c52hh"] Oct 30 21:46:57.123031 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.122345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.181045 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.180979 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664c5b7ff9-c52hh"] Oct 30 21:46:57.228064 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228207 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228207 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228207 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228207 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.228403 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.228292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.285053 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.285024 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:46:57.288563 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.288539 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf"] Oct 30 21:46:57.291172 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.291152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.329269 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329363 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329363 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329363 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329363 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329363 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329544 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.329544 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.329412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.330126 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.330100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.330219 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.330126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.330806 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.330779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.331004 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.330944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.331409 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.331375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.333292 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.333242 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf"] Oct 30 21:46:57.334729 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.334643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.335306 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.335284 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.348036 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.347997 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:46:57.348036 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.348005 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:46:57.348701 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.348212 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1papng1cqcjqm\"" Oct 30 21:46:57.348701 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.348280 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:46:57.348701 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.348368 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:46:57.348701 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.348421 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kp2zt\"" Oct 30 21:46:57.397891 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.397830 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.430379 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb95c8ea-e1a6-40f5-963f-19005925a1f8-audit-log\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.430502 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jht\" (UniqueName: \"kubernetes.io/projected/bb95c8ea-e1a6-40f5-963f-19005925a1f8-kube-api-access-c9jht\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.430502 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-metrics-server-audit-profiles\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.430502 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-client-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.430649 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-client-certs\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.430649 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.430649 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.430599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-tls\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.436221 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.436038 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:46:57.531384 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-client-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.531517 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-client-certs\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.531517 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.531517 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-tls\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.531678 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb95c8ea-e1a6-40f5-963f-19005925a1f8-audit-log\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.531678 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jht\" (UniqueName: \"kubernetes.io/projected/bb95c8ea-e1a6-40f5-963f-19005925a1f8-kube-api-access-c9jht\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.531678 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.531604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-metrics-server-audit-profiles\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.532347 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.532319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb95c8ea-e1a6-40f5-963f-19005925a1f8-audit-log\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.532737 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.532684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-metrics-server-audit-profiles\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.532737 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.532684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.537760 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.537738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-tls\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.537839 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.537743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-client-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.539826 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.539806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-client-certs\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.596188 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.596156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jht\" (UniqueName: \"kubernetes.io/projected/bb95c8ea-e1a6-40f5-963f-19005925a1f8-kube-api-access-c9jht\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.605355 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.605331 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:46:57.830930 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.830897 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664c5b7ff9-c52hh"] Oct 30 21:46:57.831369 ip-10-0-133-91 kubenswrapper[2578]: W1030 21:46:57.831338 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c952c6b_2c24_44bf_9ed2_197be86de9f8.slice/crio-52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b WatchSource:0}: Error finding container 52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b: Status 404 returned error can't find the container with id 52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b Oct 30 21:46:57.995845 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:57.994516 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf"] Oct 30 21:46:58.808144 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:58.808069 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" event={"ID":"bb95c8ea-e1a6-40f5-963f-19005925a1f8","Type":"ContainerStarted","Data":"364319ca81bb98e3539126de2b25e0968666bda4b48b0bcadfe73240078f6245"} Oct 30 21:46:58.810006 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:58.809976 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664c5b7ff9-c52hh" event={"ID":"9c952c6b-2c24-44bf-9ed2-197be86de9f8","Type":"ContainerStarted","Data":"fde4530628e8ed01e9ba8914e744388a07aebf2524680a185ceede4e9d8239a6"} Oct 30 21:46:58.810122 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:58.810025 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664c5b7ff9-c52hh" event={"ID":"9c952c6b-2c24-44bf-9ed2-197be86de9f8","Type":"ContainerStarted","Data":"52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b"} Oct 30 21:46:58.812412 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:58.812386 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"d9a4314d0e68ef55aaeb5b08307496153b34277f212ec5e192e5a4b1d0964689"} Oct 30 21:46:58.812494 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:58.812451 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"7ed34a545f9f7de2c6ddd9d7058ac58bae8b5d391bdfeae47e6517a9b3a10e1a"} Oct 30 21:46:58.812494 ip-10-0-133-91 kubenswrapper[2578]: I1030 21:46:58.812478 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"47ef3416dee572ae24f968f8745fa7f3a8932aef172f2a60e15739740e7514f4"} Oct 30 21:47:00.502445 ip-10-0-133-91 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:47:00.520530 ip-10-0-133-91 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:47:00.520784 ip-10-0-133-91 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:47:00.520847 ip-10-0-133-91 systemd[1]: kubelet.service: Consumed 3.730s CPU time. Oct 30 21:47:00.536480 ip-10-0-133-91 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:47:00.679359 ip-10-0-133-91 kubenswrapper[5607]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:47:00.679359 ip-10-0-133-91 kubenswrapper[5607]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:47:00.679359 ip-10-0-133-91 kubenswrapper[5607]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:47:00.679359 ip-10-0-133-91 kubenswrapper[5607]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:47:00.687354 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687236 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687378 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687387 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687394 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687399 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687404 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687409 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687413 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687418 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687423 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687428 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687432 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687436 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687441 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687445 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687450 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687454 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687459 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687463 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687467 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.687484 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687473 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687477 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687482 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687487 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687492 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687503 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687508 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687513 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687517 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687522 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687527 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687531 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687536 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687540 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687544 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687549 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687553 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687557 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687562 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.688295 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687566 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687570 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687574 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687581 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687587 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687592 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687597 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687602 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687606 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687610 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687614 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687618 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687625 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687630 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687635 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687640 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687651 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687657 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687664 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.689076 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687669 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687674 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687679 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687685 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687691 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687694 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687699 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687703 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687708 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687712 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687716 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687720 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687724 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687729 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687734 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687738 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687743 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687747 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687751 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687755 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.689976 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687760 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687764 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687768 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687773 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687777 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687782 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687788 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687792 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687798 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687803 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687807 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687812 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687816 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687821 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687825 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687830 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687835 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687839 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687845 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.690891 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687850 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687854 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687858 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687861 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687864 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687867 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687870 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687872 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.687875 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689108 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689124 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689129 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689134 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689139 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689143 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689148 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689152 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689157 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689161 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689166 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.691840 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689170 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689174 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689179 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689184 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689189 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689194 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689198 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689203 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689208 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689212 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689217 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689221 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689225 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689229 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689233 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689244 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689249 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689254 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689258 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.692773 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689263 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689267 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689272 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689277 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689281 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689285 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689290 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689294 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689298 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689302 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689307 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689311 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689315 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689319 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689323 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689328 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689332 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689337 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689341 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.693628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689345 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689349 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689353 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689358 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689362 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689366 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689370 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689375 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689379 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689383 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689388 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689392 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689396 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689401 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689406 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689414 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689418 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689422 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689427 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689431 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.694431 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689435 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689439 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689445 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689452 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689457 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689462 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689468 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689472 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689476 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689481 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689485 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689489 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689494 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689498 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689505 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689511 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689516 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689521 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689526 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.695227 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689530 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689534 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689539 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689543 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689547 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689552 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689556 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689561 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689567 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689571 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689575 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689580 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689584 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689588 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689593 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689597 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689601 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.689605 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690432 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.696047 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690442 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690446 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690451 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690456 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690460 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690467 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690471 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690477 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690482 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690488 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690493 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690498 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690503 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690507 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690511 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690516 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690520 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690524 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690529 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.696817 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690533 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690538 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690545 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690549 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690553 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690558 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690562 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690566 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690570 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690575 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690579 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690583 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690588 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690592 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690597 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690601 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690605 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690609 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690613 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.697628 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690618 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690622 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690628 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690632 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690636 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690641 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690645 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690649 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690653 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690657 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690661 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690665 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690669 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690673 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690678 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690684 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690688 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690692 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690697 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690701 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.698379 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690705 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690709 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690713 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690718 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690722 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690727 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690731 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690735 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690740 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690744 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690748 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690752 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690756 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690761 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690768 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690774 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690780 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690785 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690789 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.699225 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690793 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690797 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690801 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690806 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690810 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690814 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690818 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690822 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690828 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690832 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690836 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690841 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690845 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690849 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690856 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690860 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690864 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690868 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690872 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690876 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.700042 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690881 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690885 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690889 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690892 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690897 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690901 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690905 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.690910 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691066 5607 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691079 5607 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691087 5607 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691095 5607 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691102 5607 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691107 5607 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691114 5607 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691120 5607 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691126 5607 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691131 5607 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691140 5607 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691145 5607 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691151 5607 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691156 5607 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691161 5607 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:47:00.700829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691166 5607 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691171 5607 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691176 5607 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691183 5607 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691188 5607 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691192 5607 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691197 5607 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691203 5607 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691209 5607 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691214 5607 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691219 5607 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691225 5607 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691230 5607 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691235 5607 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691239 5607 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691244 5607 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691249 5607 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691256 5607 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691262 5607 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691267 5607 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691272 5607 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691276 5607 flags.go:64] FLAG: --enable-server="true" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691281 5607 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691287 5607 flags.go:64] FLAG: --event-burst="100" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691292 5607 flags.go:64] FLAG: --event-qps="50" Oct 30 21:47:00.701795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691297 5607 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691302 5607 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691310 5607 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691316 5607 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691323 5607 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691328 5607 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691333 5607 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691338 5607 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691342 5607 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691347 5607 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691352 5607 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691357 5607 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691362 5607 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691367 5607 flags.go:64] FLAG: --feature-gates="" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691373 5607 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691378 5607 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691383 5607 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691388 5607 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691393 5607 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691397 5607 flags.go:64] FLAG: --help="false" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691403 5607 flags.go:64] FLAG: --hostname-override="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691408 5607 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691413 5607 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:47:00.702796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691418 5607 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691424 5607 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691430 5607 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691435 5607 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691439 5607 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691444 5607 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691449 5607 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691454 5607 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691459 5607 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691463 5607 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691468 5607 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691474 5607 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691479 5607 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691485 5607 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691490 5607 flags.go:64] FLAG: --lock-file="" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691495 5607 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691499 5607 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691504 5607 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691514 5607 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691518 5607 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691524 5607 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691529 5607 flags.go:64] FLAG: --logging-format="text" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691533 5607 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691539 5607 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:47:00.703810 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691543 5607 flags.go:64] FLAG: --manifest-url="" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691548 5607 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691555 5607 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691559 5607 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691565 5607 flags.go:64] FLAG: --max-pods="110" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691570 5607 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691575 5607 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691580 5607 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691584 5607 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691589 5607 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691594 5607 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691600 5607 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691611 5607 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691617 5607 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691621 5607 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691626 5607 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691631 5607 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691636 5607 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691641 5607 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691647 5607 flags.go:64] FLAG: --port="10250" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691652 5607 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691658 5607 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0db8354240741ede1" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691663 5607 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691668 5607 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691673 5607 flags.go:64] FLAG: --register-node="true" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691678 5607 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:47:00.704477 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691684 5607 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691688 5607 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691693 5607 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691698 5607 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691704 5607 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691709 5607 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691714 5607 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691719 5607 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691724 5607 flags.go:64] FLAG: --runonce="false" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691729 5607 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691733 5607 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691738 5607 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691743 5607 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691747 5607 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691752 5607 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691757 5607 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691762 5607 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691767 5607 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691772 5607 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691777 5607 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691781 5607 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691786 5607 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691791 5607 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691797 5607 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691806 5607 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:47:00.705251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691813 5607 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691818 5607 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691826 5607 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691830 5607 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691835 5607 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691839 5607 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691844 5607 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691848 5607 flags.go:64] FLAG: --v="2" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691854 5607 flags.go:64] FLAG: --version="false" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691860 5607 flags.go:64] FLAG: --vmodule="" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691866 5607 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.691871 5607 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692121 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692129 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692135 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692140 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692145 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692149 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692154 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692158 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692162 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692166 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.706271 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692171 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692175 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692179 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692183 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692188 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692192 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692197 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692201 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692206 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692210 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692215 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692219 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692227 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692231 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692235 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692240 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692244 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692248 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692252 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.707247 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692256 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692261 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692265 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692269 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692273 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692278 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692282 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692286 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692292 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692298 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692302 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692307 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692311 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692316 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692320 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692324 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692328 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692332 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692338 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.708146 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692342 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692346 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692350 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692355 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692359 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692363 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692369 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692373 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692377 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692381 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692386 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692390 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692394 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692399 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692403 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692407 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692412 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692416 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692420 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692424 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.708963 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692428 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692433 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692437 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692441 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692445 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692449 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692453 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692457 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692461 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692465 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692469 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692474 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692479 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692483 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692488 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692492 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692496 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692500 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692506 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.709872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692511 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692515 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692520 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692524 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692528 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692532 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692536 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692541 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692545 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692549 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692553 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692557 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692562 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692566 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692570 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692576 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692581 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692586 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.710696 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692590 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.692633 5607 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692691 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692699 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692705 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692710 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692714 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692718 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692725 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692731 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692735 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692740 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692744 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692748 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692754 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692758 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.711489 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692763 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692767 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692770 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692775 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692779 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692783 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692786 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692791 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692795 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692799 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692803 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692807 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692811 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692816 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692820 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692825 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692829 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692833 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692838 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692842 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.712244 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692846 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692850 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692855 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692859 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692866 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692871 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692876 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692881 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692885 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692889 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692893 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692898 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692902 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692909 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692915 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692920 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692924 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692929 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692933 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.713064 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692938 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692942 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692946 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692969 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692974 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692979 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692983 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692988 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692992 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.692996 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693000 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693005 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693009 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693013 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693017 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693022 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693026 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693032 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693037 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693043 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.713998 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693047 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693052 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693056 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693060 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693065 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693069 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693074 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693078 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693082 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693086 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693090 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693095 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693099 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693103 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693107 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693111 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693117 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693122 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693126 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.714870 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693130 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693134 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693138 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693143 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693147 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693151 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693155 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693159 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693164 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693168 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693174 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693179 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693183 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.693188 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.704241 5607 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.704264 5607 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.704291 5607 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.704303 5607 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704392 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.715733 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704400 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704405 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704410 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704415 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704420 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704424 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704428 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704431 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704435 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704440 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704443 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704447 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704452 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704457 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704462 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704467 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704472 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704476 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704481 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704485 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.716625 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704490 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704496 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704502 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704507 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704514 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704521 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704526 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704534 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704539 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704543 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704548 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704552 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704557 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704562 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704566 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704571 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704575 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704581 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704587 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.717495 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704593 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704597 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704601 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704605 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704609 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704613 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704617 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704622 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704628 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704632 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704637 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704643 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704647 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704652 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704656 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704661 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704665 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704670 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704674 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704680 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.718299 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704685 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704689 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704693 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704698 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704701 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704706 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704710 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704715 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704720 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704724 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704728 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704732 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704738 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704742 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704747 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704752 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704757 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704761 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704767 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704771 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.719170 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704776 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704780 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704784 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704789 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704793 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704798 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704802 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704807 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704811 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704816 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704821 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704825 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704829 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704834 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704839 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704844 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704848 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704853 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704857 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.720062 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704861 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704865 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704870 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704876 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704880 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704885 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.704889 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.704935 5607 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705183 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705194 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705198 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705202 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705206 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705211 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705215 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.720809 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705221 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705226 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705230 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705234 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705247 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705252 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705260 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705267 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705271 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705277 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705282 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705286 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705290 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705293 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705296 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705299 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705301 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705304 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705307 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.721467 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705309 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705312 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705314 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705317 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705320 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705324 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705328 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705333 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705338 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705342 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705345 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705349 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705351 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705354 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705360 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705367 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705372 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705378 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705382 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705385 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.722344 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705389 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705394 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705398 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705402 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705406 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705409 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705411 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705413 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705417 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705419 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705422 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705425 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705429 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705432 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705436 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705441 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705446 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705450 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705455 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705460 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.723282 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705464 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705468 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705472 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705476 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705482 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705486 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705491 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705495 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705499 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705503 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705507 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705512 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705516 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705520 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705524 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705528 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705532 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705537 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705541 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705546 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.724175 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705551 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705556 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705561 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705565 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705569 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705574 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705578 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705580 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705583 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705588 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705592 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705596 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705600 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705604 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705609 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705613 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705618 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705623 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705627 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.725022 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705631 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.705676 5607 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705722 5607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705730 5607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705736 5607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705742 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705748 5607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705753 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705757 5607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705761 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705765 5607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705770 5607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705773 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705776 5607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705780 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705784 5607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:47:00.725844 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705788 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705792 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705795 5607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705797 5607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705801 5607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705806 5607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705810 5607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705814 5607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705818 5607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705823 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705828 5607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705832 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705834 5607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705837 5607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705841 5607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705847 5607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705851 5607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705855 5607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705857 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705860 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:47:00.726542 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705863 5607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705866 5607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705869 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705871 5607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705873 5607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705877 5607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705880 5607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705884 5607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705889 5607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705892 5607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705895 5607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705898 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705901 5607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705905 5607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705909 5607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705913 5607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705918 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705928 5607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705932 5607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:47:00.727475 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705937 5607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705941 5607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705944 5607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.705995 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706003 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706007 5607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706012 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706016 5607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706020 5607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706027 5607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706034 5607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706040 5607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706045 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706049 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706053 5607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706058 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706062 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706066 5607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706070 5607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:47:00.728348 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706074 5607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706078 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706082 5607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706087 5607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706091 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706095 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706099 5607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706103 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706108 5607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706112 5607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706116 5607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706123 5607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706127 5607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706131 5607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706135 5607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706139 5607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706143 5607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706147 5607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706155 5607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706160 5607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:47:00.729168 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706166 5607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706172 5607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706176 5607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706181 5607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706185 5607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706189 5607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706193 5607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706197 5607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706202 5607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706206 5607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706210 5607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706214 5607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706218 5607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:00.706222 5607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.706313 5607 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.707801 5607 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.707881 5607 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.708168 5607 server.go:1014] "Starting client certificate rotation" Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.708859 5607 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:47:00.729899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.708972 5607 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.709057 5607 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:40:49 +0000 UTC" deadline="2028-05-19 23:49:01.250925113 +0000 UTC" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.709105 5607 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13610h2m0.541824349s" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.710552 5607 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.712386 5607 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.715553 5607 log.go:25] "Validated CRI v1 image API" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.717321 5607 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:47:00.730656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.721574 5607 fs.go:135] Filesystem UUIDs: map[26f7696d-b143-43db-ae3f-ca776eea8b11:/dev/nvme0n1p4 708024d4-dc90-4ff5-b922-57cad575a694:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:47:00.731121 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.721619 5607 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/364319ca81bb98e3539126de2b25e0968666bda4b48b0bcadfe73240078f6245/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/364319ca81bb98e3539126de2b25e0968666bda4b48b0bcadfe73240078f6245/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0140305c-d4d1-4ae9-877a-6c6d21b80d0d/volumes/kubernetes.io~projected/kube-api-access-4vl64:{mountpoint:/var/lib/kubelet/pods/0140305c-d4d1-4ae9-877a-6c6d21b80d0d/volumes/kubernetes.io~projected/kube-api-access-4vl64 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0140305c-d4d1-4ae9-877a-6c6d21b80d0d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0140305c-d4d1-4ae9-877a-6c6d21b80d0d/volumes/kubernetes.io~secret/cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volumes/kubernetes.io~projected/kube-api-access-5dhtv:{mountpoint:/var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volumes/kubernetes.io~projected/kube-api-access-5dhtv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/191e72fd-de5a-4dea-9592-c75367da8608/volumes/kubernetes.io~projected/kube-api-access-55kb5:{mountpoint:/var/lib/kubelet/pods/191e72fd-de5a-4dea-9592-c75367da8608/volumes/kubernetes.io~projected/kube-api-access-55kb5 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bdaec03-c2f8-446e-9376-bed9a55677c9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1bdaec03-c2f8-446e-9376-bed9a55677c9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/248fb46a-f3b1-4d89-9d88-0191280d983d/volumes/kubernetes.io~projected/kube-api-access-pjsts:{mountpoint:/var/lib/kubelet/pods/248fb46a-f3b1-4d89-9d88-0191280d983d/volumes/kubernetes.io~projected/kube-api-access-pjsts major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/285863f3-fa2b-4d9a-80b2-7421899e97c4/volumes/kubernetes.io~projected/kube-api-access-2cw9d:{mountpoint:/var/lib/kubelet/pods/285863f3-fa2b-4d9a-80b2-7421899e97c4/volumes/kubernetes.io~projected/kube-api-access-2cw9d major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~projected/kube-api-access-nkgds:{mountpoint:/var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~projected/kube-api-access-nkgds major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35cd7059-12c6-4c41-99ef-86eea5a5873e/volumes/kubernetes.io~projected/kube-api-access-8qt92:{mountpoint:/var/lib/kubelet/pods/35cd7059-12c6-4c41-99ef-86eea5a5873e/volumes/kubernetes.io~projected/kube-api-access-8qt92 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35cd7059-12c6-4c41-99ef-86eea5a5873e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/35cd7059-12c6-4c41-99ef-86eea5a5873e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~projected/kube-api-access-7k6bl:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~projected/kube-api-access-7k6bl major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f4e46f2-7b2a-4518-861c-fb94540b0231/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/6f4e46f2-7b2a-4518-861c-fb94540b0231/volumes/kubernetes.io~secret/tls-certificates major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/729a10ce-45d9-4094-8f2b-c5af2b19d159/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/729a10ce-45d9-4094-8f2b-c5af2b19d159/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80e5d657-00d6-433e-b9bd-0ca172deb64d/volumes/kubernetes.io~projected/kube-api-access-gg5bq:{mountpoint:/var/lib/kubelet/pods/80e5d657-00d6-433e-b9bd-0ca172deb64d/volumes/kubernetes.io~projected/kube-api-access-gg5bq major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80e5d657-00d6-433e-b9bd-0ca172deb64d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/80e5d657-00d6-433e-b9bd-0ca172deb64d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8639240c-cb7f-45b0-a0ce-97eb850716cc/volumes/kubernetes.io~projected/kube-api-access-skhxz:{mountpoint:/var/lib/kubelet/pods/8639240c-cb7f-45b0-a0ce-97eb850716cc/volumes/kubernetes.io~projected/kube-api-access-skhxz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~projected/kube-api-access-wd9xw:{mountpoint:/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~projected/kube-api-access-wd9xw major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4/volumes/kubernetes.io~projected/kube-api-access-prh52:{mountpoint:/var/lib/kubelet/pods/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4/volumes/kubernetes.io~projected/kube-api-access-prh52 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e/volumes/kubernetes.io~projected/kube-api-access-hxksq:{mountpoint:/var/lib/kubelet/pods/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e/volumes/kubernetes.io~projected/kube-api-access-hxksq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~projected/kube-api-access-j8dvx:{mountpoint:/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~projected/kube-api-access-j8dvx major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8dbcc04-78d4-4ac0-8697-deec99270f95/volumes/kubernetes.io~projected/kube-api-access-9hdxt:{mountpoint:/var/lib/kubelet/pods/a8dbcc04-78d4-4ac0-8697-deec99270f95/volumes/kubernetes.io~projected/kube-api-access-9hdxt major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~projected/kube-api-access-vcbwm:{mountpoint:/var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~projected/kube-api-access-vcbwm major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/kube-api-access-txcwv:{mountpoint:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/kube-api-access-txcwv major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/registry-tls major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~projected/kube-api-access-c9jht:{mountpoint:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~projected/kube-api-access-c9jht major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb385ed2-d5f0-4025-b614-1db5e6f0dd68/volumes/kubernetes.io~projected/kube-api-access-k75sw:{mountpoint:/var/lib/kubelet/pods/cb385ed2-d5f0-4025-b614-1db5e6f0dd68/volumes/kubernetes.io~projected/kube-api-access-k75sw major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a/volumes/kubernetes.io~projected/kube-api-access-5cd9p:{mountpoint:/var/lib/kubelet/pods/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a/volumes/kubernetes.io~projected/kube-api-access-5cd9p major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~projected/kube-api-access-dqkft:{mountpoint:/var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~projected/kube-api-access-dqkft major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:314 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9c15a1a98199df1080d70f50f52fc96d9c32501fea2ae20c8c796fce9b0f32e1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/047fced20dbb9b445be16e4d60ce5503f7e17d5d7e3e0249494e303b17981d92/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/160a971668c508c96294f2f07224f1543a8669a67aa37e49fa82c5b81fd3fff8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/192da581560d5cd956abc225895aded813600cf02f55a98eae58b8d1a8d00931/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c25520a332aac4e971c7dd0cd07c3fce321381845118ab660114f2911325aff8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f9b7f8b891b5dcab6b55f9019864ace12a91f5e31f073f51d160a8bae8ab1283/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7ec3f3d7871f1f7416b97b7412f8f07b0fb66ee748ce9e6f15e324ed2eaa0037/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c7ef4228e2c5215f9c6f256f6dc0c5a34b9d00202eb002cb01f1756b9f867898/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a1bbc7b1ca282b58f9f8e11844fead66c12443be5f7572342a9627afbeea18af/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e0fe3940479201947498b3517845c231538ddbe6e42c0d1ff193320c82187d94/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/6afc758cddd0a47765281e7d1c1a7c1f891c5ca6b43aba4358562498cfb71a10/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/398dfdac8d5ca72abd0610703c8c6d40c3a89610dce765fc80f3726112ee8e6f/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/dc3311f611772f8a5f6cb72fdb24ba324f7a6cb92f953a42c6ccdc7e08d0cbe1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/11ca15cde259608803aed221f0e6854a9112ca36c567a77c1da79fab63e49971/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3e4899a7d9cf6b4d7b9d855c232756bda456fc94fb2b87b9b3b9d4b6860ae9ee/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/88947723c3f8d12194a92b8c9901a2d763f78fe2a1e604f8be7596efb9866168/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6dc618b3e1057c327ac96c82c3c0c7ce2f1cb0e01b75750bdc7e1bec975c9591/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6f0b2dca556f74ebeac8f4a13f88e4b7fb07cba47d73b2c6643e022e10578f5d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/27dfd88c8edf09b7f27e1de3137db8d7a85b40b803bfcc83adcf14e514469a57/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/045bb81984229cd362613a5db2d715be25be12bc2fabe719af9f932c62426304/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fa5dbe18e05fdea7ae82e74a1cc31941a73681f5b0d9a4a4eca07f4ee14f062a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/cfff2a12c5f39e640c7eae35092f2de5da6d3dc35a22b0ea6397d0d06ad3908b/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/e89491a16fac683e4bf6e870ab464faf2a6b99eae091e86431cfb83c2f562e47/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/606175d1c24624fb3b0a71b9490e2a3b76bff0163cb2e10ad5bb8e863549cdac/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b517b81838f1aa4ad99befdbe2ec8d09d80d8324385a038cc66eda858e701a7b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ce9034914f6bd44de8357ff0b7c1270ea6d0240c438b3d7eeadc1663c848b2f2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/72b0a125c165b77625b520c469041303cf00fd6eebc826b2e805e99c9b95e924/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/96bf86fe08327cf9db2d1fcbf9c375ade04e669f2607c28da9de734bbede35b4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b40b6a2af5ccef562f6c8c70dce55b52982305e782d9a310c2db505ea16dfab5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/55d2839edc239db0a51d90c40eb0f3deb0f591c89a4617fd666f4833846b98df/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/da9ccb1e25e6e8b38dcd71bcee18fe641a396ff4faca973d0b31cda22342aa3b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/d222c9580cb8c24db6279ad7bc92076ecb4b3a8cb517a8222bc6e207f37f2bf6/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/1cfcca0bd3f766e0701f994f808e1a508ff3aa50696f3002c51aeac877c1f9be/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/dceace6eb8a0649a92ff50be6b9d67c467d5efb6b06c67f74f64dd00a9c0f967/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b77078f9506e09abae27ba972290cce90bb56a01163dab3f8bf0f1d53ad7102b/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/d84e751208382df8ea2e14f113acf84ddea365c01f1141af228444f4b1362569/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/3a0d9434d33d923ad51d37c9b007d0841283778cc7ebea6f1db91cf3663878bb/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/5868d3a0f2b8b8563639053a79e0172117c13d44919c4c43185673937b1be733/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/59cc5527ac1a143df975184a55758144e1071dbb8e8877012da205102d592291/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/b965d8f5e8e3115111b56e93a5fb8ac952f94844c21bfd8d4caf5f6ce730b7a6/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/54264efa548b2e64688c0791ca492ba570f49ed386826bc374221543d5c513f6/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/5f4a915dafa7afebc5d2239ba513150f8382c5133dc9aabf6e2fafd66de9413a/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/f6666dcfd728b4eac2a3c57e1498d6bc542620d072e37e6f9c5942548b6be910/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/401f8200003cc6938b6ab3f57c579c904ecb03e29ff9d9c4230cd519eb714495/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/519a8ff48f6218f2929e1c1d8a41be63e906d16eaebb684fd2b21a0c1e0a9ded/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/866961ee289387dfb7a0688efcd17a18f9dc0206837cef1a902d286dd3aabcd7/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/3406b5cdd2af1bb5fccebc23d291da9448938fc3d9e4ea8d290e0507b98c4b9a/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/06550eb2bbbd63068439704836bd65833073b0333b5a0b583b4d3df84805287e/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/8f5afb7aea0e059d715be1b2c03f1082897f7b15ad8478bd297bb2d9010cc6ea/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/8c82aa3e1a0070dfdf0331e4148a29fc111b21750a3de737d932098eb0dea4ee/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/fa128a74bd24679be747bbebdac68024e3171f63c11d42efaf21db5c3f8a3886/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/34a1ecdfa0c221e1d280407ebcee118aed8ddb7ba0b94d5b29706807db853fc0/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/54133091247a84b4699a9179a5ca2fa19b88c570863b21cd57496824045db1e2/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/d8b6342ed3787d8524ac7772fe133e77f93ffed28e9cd9cbbf8de8be6b41ef7e/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/50896aea3b2678dee556a2c14691c4dec302462410829fef65bc86f49b22ea1c/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/692c7ec9d3672382a6c2a9e42caaa4cb772c2d9d6d55a3b3a748dc8c771be171/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/29db931729549552f32a872b83484f3f698bf83f817b961e4c813f55ff53aed9/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/b0001804e675fc6194b4314f7d30dc5d6a2966daf3dc0604e725c5a5926e60b6/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/d6c7c8c20dfb44daa116e15e694c54659d2c520458191d3badee9a383525250f/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/d52315dbddd8732c4bbc3ee6b8a98c4ca82e108018b2c7059ce1c1f0cb3c3540/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/dbb39a395dbb7c40761845345e00cadb40e779558be937ffccf1bd21a4cc8950/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dbd0cfb57f15f5b8776a2c7be2dc01fd352f8f3f519bc36dc0699937b2a9ff8c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/baa49ee1656888f476ec5dcbc52899097ea8255d091b49803d8efe3b3fbf8c3e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/62fd0ed8d9739db618aa888b0e95a0bee1d41f0e2d5f2af312d5e3c37ef6a375/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a8af98f42dc32ecf612df6028651f5b3c63a9ca360b211e03a1c186aa10a0d54/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/54f2ac025ef5d7379162083912225620d757a2addb3968e45c87eca25dfd6c64/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/085dff2156478aad9d0d8c6e83f0373b34b7b86d982b5f21ff1acb010a642d9d/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ca77fb4f627ccb419d0ee79d8f62469a0e1669db6bb2cf4ab4bcc8d731714390/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/35d941dbe78af53c1a746a7ece453b609b1a6f2491ea4d680173e42834e618fb/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:47:00.739137 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.738447 5607 manager.go:217] Machine: {Timestamp:2026-10-30 21:47:00.737683714 +0000 UTC m=+0.138239853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099598 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25096ae0b08b28544b251e8de6ab5d SystemUUID:ec25096a-e0b0-8b28-544b-251e8de6ab5d BootID:d8ba97bb-01b3-4da1-928c-7865669161ff Filesystems:[{Device:/run/containers/storage/overlay-containers/a4fe5df17427f4f7759fc8a91a85fa5a77de98d50402c51d6da55a8f2d83cbf2/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c88bfeadd1a5cd6e307d036bcfe7e9b1e98279c481fefa37168fdd6a7bcf2c31/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a/volumes/kubernetes.io~projected/kube-api-access-5cd9p DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/248fb46a-f3b1-4d89-9d88-0191280d983d/volumes/kubernetes.io~projected/kube-api-access-pjsts DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54a110f16afe14b9e270865f56b10ab3cec27b56bca68ce6e53c7b2c4004aa5b/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bdaec03-c2f8-446e-9376-bed9a55677c9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8639240c-cb7f-45b0-a0ce-97eb850716cc/volumes/kubernetes.io~projected/kube-api-access-skhxz DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35cd7059-12c6-4c41-99ef-86eea5a5873e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adf9b329045cf8e6d018ce608fd722dc77a3b2b3842fe65072c2c61a49ac2b85/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80e5d657-00d6-433e-b9bd-0ca172deb64d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0140305c-d4d1-4ae9-877a-6c6d21b80d0d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:378 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~projected/kube-api-access-wd9xw DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f7848d0985e71d5ea6e5a0351f416dd5d9e52d828bfbf8e55301deae7604830/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/285863f3-fa2b-4d9a-80b2-7421899e97c4/volumes/kubernetes.io~projected/kube-api-access-2cw9d DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b48f1dce5f6f3ba5e570aac33d7b5f8dfe80dd902e595d75b9dd4032a94051a1/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/191e72fd-de5a-4dea-9592-c75367da8608/volumes/kubernetes.io~projected/kube-api-access-55kb5 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f23749313293a1da32f99a9934d0ba3d4417c35c91aaf97d599de9caaa80b50/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:371 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/kube-api-access-txcwv DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:377 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e/volumes/kubernetes.io~projected/kube-api-access-hxksq DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:367 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/729a10ce-45d9-4094-8f2b-c5af2b19d159/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb9e7bada53204f9b86c025a872fb9c2a7f2d4f0e5f2a00ac9bfae3b040230c8/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f4e46f2-7b2a-4518-861c-fb94540b0231/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb385ed2-d5f0-4025-b614-1db5e6f0dd68/volumes/kubernetes.io~projected/kube-api-access-k75sw DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~projected/kube-api-access-dqkft DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~projected/kube-api-access-vcbwm DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:363 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~projected/kube-api-access-nkgds DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c975e8e95ba382de34b0811fca52592df9826f6c382cf4af008f9b037c8b5ba/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3eca9620e4633882b58807653d7c2df99c331f8acf6c3a267a4dc87dfd0a8bb8/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:368 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes/kubernetes.io~projected/kube-api-access-j8dvx DeviceMajor:0 DeviceMinor:379 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0140305c-d4d1-4ae9-877a-6c6d21b80d0d/volumes/kubernetes.io~projected/kube-api-access-4vl64 DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e9328b7-cd5f-4a63-8895-be053051f188/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3be81068d35548e45ec0f16f52e4280407854410d5e5099cff5d57a2a6130754/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80e5d657-00d6-433e-b9bd-0ca172deb64d/volumes/kubernetes.io~projected/kube-api-access-gg5bq DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a8dbcc04-78d4-4ac0-8697-deec99270f95/volumes/kubernetes.io~projected/kube-api-access-9hdxt DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f681716ffa0cbed0009a0175a0b0176ec6329c677a5f4fe4c2678361ed6f4a5/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df17e379a5e1ebb0009aafb636684becf3672f91b0be79d06131f43d063b5297/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab980c40-b10c-4f42-90d8-bd930becdb72/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee85c5a862b3e4bb353d197a48d357f9ea997013de408f60cbb863bfa3595c4d/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/18cf0a1a-efb8-4aef-ac2a-16e294018ccf/volumes/kubernetes.io~projected/kube-api-access-5dhtv DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4/volumes/kubernetes.io~projected/kube-api-access-prh52 DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e61ec43826c8fd6a82f20b26a3e173e9fcd134b6b79b82cdb4a000489682db9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1a612be2c795e474524eb44174f38c6edba75168fcaf4b98a3c13659df0c298/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/364319ca81bb98e3539126de2b25e0968666bda4b48b0bcadfe73240078f6245/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bcaefea7506aa32e742daab8cd990d2d666e28dd0c934a3e2fdfc11c2df865b/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:382 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8392e7803c5d0fc37c8cdf4a46e10f749797b4e7f82a38bc2e01bdd3caf7a03/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffe895c1-e716-45d5-94eb-58668c4bd1c1/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e9e7e1c37993c1f75f0810436c8bbfef5887073ecbe188d7bb2405a61da6487/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5619b33-0539-4471-ac48-e5f51da5ab38/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa97d352a593bf88f63e5d9bf7df61d44089830d452850acb78433cf0cca64cb/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ebfccd4-cbd5-4319-be94-3da3df1d8566/volumes/kubernetes.io~projected/kube-api-access-7k6bl DeviceMajor:0 DeviceMinor:372 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bb95c8ea-e1a6-40f5-963f-19005925a1f8/volumes/kubernetes.io~projected/kube-api-access-c9jht DeviceMajor:0 DeviceMinor:385 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35cd7059-12c6-4c41-99ef-86eea5a5873e/volumes/kubernetes.io~projected/kube-api-access-8qt92 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:364319ca81bb98e MacAddress:9a:b6:cc:40:27:e8 Speed:10000 Mtu:8901} {Name:3eca9620e463388 MacAddress:7a:f0:8f:be:38:4f Speed:10000 Mtu:8901} {Name:4f7848d0985e71d MacAddress:7a:06:68:db:eb:16 Speed:10000 Mtu:8901} {Name:52e9d887a06276b MacAddress:d6:fc:2e:61:fe:cb Speed:10000 Mtu:8901} {Name:8486dbc43f8a2e4 MacAddress:fe:30:20:83:93:8e Speed:10000 Mtu:8901} {Name:8bcaefea7506aa3 MacAddress:e6:37:5d:85:9e:a5 Speed:10000 Mtu:8901} {Name:9f23749313293a1 MacAddress:4a:08:98:3d:99:e1 Speed:10000 Mtu:8901} {Name:aa97d352a593bf8 MacAddress:66:b0:d3:f5:df:ca Speed:10000 Mtu:8901} {Name:adf9b329045cf8e MacAddress:f6:6d:1f:87:ac:23 Speed:10000 Mtu:8901} {Name:b48f1dce5f6f3ba MacAddress:66:2d:dc:9b:16:94 Speed:10000 Mtu:8901} {Name:bb9e7bada53204f MacAddress:06:9f:91:76:92:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f7:f2:00:db Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:26:1d:45:af:ff Speed:0 Mtu:8901} {Name:c1a612be2c795e4 MacAddress:f2:7d:dd:7d:95:cb Speed:10000 Mtu:8901} {Name:df17e379a5e1ebb MacAddress:f6:74:a8:de:26:8e Speed:10000 Mtu:8901} {Name:ee85c5a862b3e4b MacAddress:de:7e:7b:0c:c8:91 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f7:f2:00:db Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1e:11:3c:f4:9b:48 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:e0:34:7d:98:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:47:00.739663 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.739512 5607 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:47:00.739663 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.739581 5607 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740274 5607 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740309 5607 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-91.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740505 5607 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740524 5607 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740544 5607 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740568 5607 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:47:00.740906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.740720 5607 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:47:00.741490 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.741456 5607 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:47:00.741800 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.741737 5607 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:47:00.742890 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.742872 5607 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:47:00.743002 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.742894 5607 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:47:00.743002 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.742919 5607 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:47:00.743002 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.742934 5607 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:47:00.753129 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.751835 5607 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:47:00.753129 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.752765 5607 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:47:00.753129 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.752778 5607 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:47:00.753557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.753512 5607 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:47:00.767433 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767286 5607 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:47:00.767433 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767303 5607 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:47:00.767433 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767427 5607 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:47:00.767755 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767691 5607 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:47:00.767755 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767725 5607 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:47:00.767850 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767838 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:47:00.767896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767851 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:47:00.767896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767860 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:47:00.767896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767877 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:47:00.767896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767887 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:47:00.767896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767896 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767907 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767917 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767929 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767939 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.767995 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.768010 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.768041 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:47:00.768179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.768053 5607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:47:00.772394 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.772374 5607 server.go:1263] "Started kubelet" Oct 30 21:47:00.772989 ip-10-0-133-91 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:47:00.776063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.773915 5607 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:47:00.776063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.774017 5607 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:47:00.776063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.774094 5607 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:47:00.776063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.774499 5607 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:47:00.776063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.775179 5607 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:47:00.786107 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786011 5607 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:47:00.786203 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786147 5607 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:47:00.786203 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786193 5607 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:50 +0000 UTC" deadline="2028-06-22 02:03:49.059916804 +0000 UTC" Oct 30 21:47:00.786317 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786210 5607 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14404h16m48.273708975s" Oct 30 21:47:00.786317 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786248 5607 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:47:00.786317 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786255 5607 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:47:00.786468 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.786353 5607 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:47:00.789629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789522 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" volumeName="kubernetes.io/projected/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-kube-api-access-5dhtv" seLinuxMountContext="" Oct 30 21:47:00.789629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789558 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" volumeName="kubernetes.io/secret/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:47:00.789629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789571 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="191e72fd-de5a-4dea-9592-c75367da8608" volumeName="kubernetes.io/projected/191e72fd-de5a-4dea-9592-c75367da8608-kube-api-access-55kb5" seLinuxMountContext="" Oct 30 21:47:00.789629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789587 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/empty-dir/3ebfccd4-cbd5-4319-be94-3da3df1d8566-tmp" seLinuxMountContext="" Oct 30 21:47:00.789629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789600 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789636 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789650 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789663 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789677 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789689 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789701 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd3fa78-0fa0-41de-b566-bc0cea25b5b4" volumeName="kubernetes.io/empty-dir/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-data-volume" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789726 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789739 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd3fa78-0fa0-41de-b566-bc0cea25b5b4" volumeName="kubernetes.io/configmap/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789750 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789764 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab980c40-b10c-4f42-90d8-bd930becdb72" volumeName="kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-metrics-client-ca" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789777 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab980c40-b10c-4f42-90d8-bd930becdb72" volumeName="kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789789 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-certificates" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789801 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="285863f3-fa2b-4d9a-80b2-7421899e97c4" volumeName="kubernetes.io/projected/285863f3-fa2b-4d9a-80b2-7421899e97c4-kube-api-access-2cw9d" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789813 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/configmap/3ebfccd4-cbd5-4319-be94-3da3df1d8566-metrics-client-ca" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789825 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789836 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f4e46f2-7b2a-4518-861c-fb94540b0231" volumeName="kubernetes.io/secret/6f4e46f2-7b2a-4518-861c-fb94540b0231-tls-certificates" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789847 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab980c40-b10c-4f42-90d8-bd930becdb72" volumeName="kubernetes.io/projected/ab980c40-b10c-4f42-90d8-bd930becdb72-kube-api-access-vcbwm" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789859 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" volumeName="kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789870 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789882 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd3fa78-0fa0-41de-b566-bc0cea25b5b4" volumeName="kubernetes.io/secret/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789893 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789905 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8dbcc04-78d4-4ac0-8697-deec99270f95" volumeName="kubernetes.io/empty-dir/a8dbcc04-78d4-4ac0-8697-deec99270f95-tmp" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789917 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789933 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-client-ca-bundle" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789944 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-trusted-ca" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789974 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789987 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="248fb46a-f3b1-4d89-9d88-0191280d983d" volumeName="kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.789998 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8dbcc04-78d4-4ac0-8697-deec99270f95" volumeName="kubernetes.io/projected/a8dbcc04-78d4-4ac0-8697-deec99270f95-kube-api-access-9hdxt" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790011 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe895c1-e716-45d5-94eb-58668c4bd1c1" volumeName="kubernetes.io/projected/ffe895c1-e716-45d5-94eb-58668c4bd1c1-kube-api-access-dqkft" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790022 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0140305c-d4d1-4ae9-877a-6c6d21b80d0d" volumeName="kubernetes.io/secret/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-cert" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790034 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="248fb46a-f3b1-4d89-9d88-0191280d983d" volumeName="kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790045 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35cd7059-12c6-4c41-99ef-86eea5a5873e" volumeName="kubernetes.io/projected/35cd7059-12c6-4c41-99ef-86eea5a5873e-kube-api-access-8qt92" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790055 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/projected/3ebfccd4-cbd5-4319-be94-3da3df1d8566-kube-api-access-7k6bl" seLinuxMountContext="" Oct 30 21:47:00.790037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790068 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80e5d657-00d6-433e-b9bd-0ca172deb64d" volumeName="kubernetes.io/secret/80e5d657-00d6-433e-b9bd-0ca172deb64d-metrics-tls" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790080 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790101 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/empty-dir/b5619b33-0539-4471-ac48-e5f51da5ab38-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790120 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb385ed2-d5f0-4025-b614-1db5e6f0dd68" volumeName="kubernetes.io/configmap/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790131 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790154 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bdaec03-c2f8-446e-9376-bed9a55677c9" volumeName="kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790167 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8639240c-cb7f-45b0-a0ce-97eb850716cc" volumeName="kubernetes.io/empty-dir/8639240c-cb7f-45b0-a0ce-97eb850716cc-tmp-dir" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790180 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790194 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab980c40-b10c-4f42-90d8-bd930becdb72" volumeName="kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-tls" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790206 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-tls" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790218 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790230 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/projected/bb95c8ea-e1a6-40f5-963f-19005925a1f8-kube-api-access-c9jht" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790242 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="248fb46a-f3b1-4d89-9d88-0191280d983d" volumeName="kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-binary-copy" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790254 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="285863f3-fa2b-4d9a-80b2-7421899e97c4" volumeName="kubernetes.io/configmap/285863f3-fa2b-4d9a-80b2-7421899e97c4-serviceca" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790265 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e9328b7-cd5f-4a63-8895-be053051f188" volumeName="kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-tmp" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790277 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790290 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe895c1-e716-45d5-94eb-58668c4bd1c1" volumeName="kubernetes.io/configmap/ffe895c1-e716-45d5-94eb-58668c4bd1c1-metrics-client-ca" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790302 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0140305c-d4d1-4ae9-877a-6c6d21b80d0d" volumeName="kubernetes.io/projected/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-kube-api-access-4vl64" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790324 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80e5d657-00d6-433e-b9bd-0ca172deb64d" volumeName="kubernetes.io/empty-dir/80e5d657-00d6-433e-b9bd-0ca172deb64d-tmp-dir" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790337 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90fc0a23-c03b-40f3-83ac-61ba7b26bc5e" volumeName="kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790348 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790360 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab980c40-b10c-4f42-90d8-bd930becdb72" volumeName="kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790373 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-kube-api-access-txcwv" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790384 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a" volumeName="kubernetes.io/projected/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kube-api-access-5cd9p" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790397 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80e5d657-00d6-433e-b9bd-0ca172deb64d" volumeName="kubernetes.io/configmap/80e5d657-00d6-433e-b9bd-0ca172deb64d-config-volume" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790419 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fd3fa78-0fa0-41de-b566-bc0cea25b5b4" volumeName="kubernetes.io/projected/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-api-access-prh52" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790433 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5619b33-0539-4471-ac48-e5f51da5ab38" volumeName="kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-bound-sa-token" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790445 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe895c1-e716-45d5-94eb-58668c4bd1c1" volumeName="kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790456 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e9328b7-cd5f-4a63-8895-be053051f188" volumeName="kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-etc-tuned" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790468 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e9328b7-cd5f-4a63-8895-be053051f188" volumeName="kubernetes.io/projected/2e9328b7-cd5f-4a63-8895-be053051f188-kube-api-access-nkgds" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790480 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790492 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="729a10ce-45d9-4094-8f2b-c5af2b19d159" volumeName="kubernetes.io/configmap/729a10ce-45d9-4094-8f2b-c5af2b19d159-konnectivity-ca" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790504 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80e5d657-00d6-433e-b9bd-0ca172deb64d" volumeName="kubernetes.io/projected/80e5d657-00d6-433e-b9bd-0ca172deb64d-kube-api-access-gg5bq" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790516 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790528 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab980c40-b10c-4f42-90d8-bd930becdb72" volumeName="kubernetes.io/empty-dir/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790540 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" volumeName="kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-env-overrides" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790552 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8639240c-cb7f-45b0-a0ce-97eb850716cc" volumeName="kubernetes.io/projected/8639240c-cb7f-45b0-a0ce-97eb850716cc-kube-api-access-skhxz" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790563 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790575 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c952c6b-2c24-44bf-9ed2-197be86de9f8" volumeName="kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790586 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/empty-dir/bb95c8ea-e1a6-40f5-963f-19005925a1f8-audit-log" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790598 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="248fb46a-f3b1-4d89-9d88-0191280d983d" volumeName="kubernetes.io/projected/248fb46a-f3b1-4d89-9d88-0191280d983d-kube-api-access-pjsts" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790610 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35cd7059-12c6-4c41-99ef-86eea5a5873e" volumeName="kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790623 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790636 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cb23260-4a0e-44e6-8023-124e58c32c5e" volumeName="kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790647 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb95c8ea-e1a6-40f5-963f-19005925a1f8" volumeName="kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790659 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" volumeName="kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-config" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790670 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="191e72fd-de5a-4dea-9592-c75367da8608" volumeName="kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-multus-daemon-config" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790681 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="191e72fd-de5a-4dea-9592-c75367da8608" volumeName="kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-cni-binary-copy" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790694 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ebfccd4-cbd5-4319-be94-3da3df1d8566" volumeName="kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790706 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="729a10ce-45d9-4094-8f2b-c5af2b19d159" volumeName="kubernetes.io/secret/729a10ce-45d9-4094-8f2b-c5af2b19d159-agent-certs" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790718 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb385ed2-d5f0-4025-b614-1db5e6f0dd68" volumeName="kubernetes.io/projected/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-kube-api-access-k75sw" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790729 5607 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe895c1-e716-45d5-94eb-58668c4bd1c1" volumeName="kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790740 5607 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.790747 5607 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:00.791066 5607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.792193 5607 factory.go:55] Registering systemd factory Oct 30 21:47:00.792947 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.792210 5607 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:47:00.798105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.798005 5607 factory.go:153] Registering CRI-O factory Oct 30 21:47:00.798105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.798024 5607 factory.go:223] Registration of the crio container factory successfully Oct 30 21:47:00.798105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.798078 5607 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:47:00.798105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.798098 5607 factory.go:103] Registering Raw factory Oct 30 21:47:00.798105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.798112 5607 manager.go:1199] Started watching for new ooms in manager Oct 30 21:47:00.798563 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.798477 5607 manager.go:319] Starting recovery of all containers Oct 30 21:47:00.848854 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.848811 5607 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:47:00.861941 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.861915 5607 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:47:00.861941 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.861946 5607 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:47:00.862165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.861995 5607 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:47:00.862165 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:00.862038 5607 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:47:00.868565 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.868541 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:47:00.889543 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.889514 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:47:00.891247 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:00.891206 5607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-91.us-west-2.compute.internal\" not found" Oct 30 21:47:00.910688 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.910655 5607 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="b9e6d102ee9925d5ed302db57a78f19b911c25dbcf91795595bc4bdd31b99892" exitCode=0 Oct 30 21:47:00.910688 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.910689 5607 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="f7d7a25b4a897a7aae3f56175e5c3ce9b37349bce69285502cafffd63f143420" exitCode=0 Oct 30 21:47:00.910858 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.910701 5607 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="ab26072c248c6c77497adc6eb63ff7d7a9f748d54a4236c565a3463ba06d8d37" exitCode=0 Oct 30 21:47:00.910858 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.910710 5607 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="a56ea0e17c21ce2cd87a68d9619f6960409e2fb1fe1b552d59e9e5324e489eac" exitCode=0 Oct 30 21:47:00.910858 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.910719 5607 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="d1f54133129018ee32dcf969c732c8c7ed8b0bb5b3c4da09a49a7ad0709c5425" exitCode=0 Oct 30 21:47:00.910858 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.910737 5607 generic.go:356] "Generic (PLEG): container finished" podID="248fb46a-f3b1-4d89-9d88-0191280d983d" containerID="1bddcdf233b296c0f426397bd672ef19bf9e36b9fc8b08e8bfea174a8ed4256d" exitCode=0 Oct 30 21:47:00.954179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.954151 5607 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:47:00.954332 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.954286 5607 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:47:00.954581 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.954565 5607 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:47:00.962151 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:00.962129 5607 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:47:00.962610 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.962357 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:47:00.963198 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.963156 5607 generic.go:356] "Generic (PLEG): container finished" podID="18cf0a1a-efb8-4aef-ac2a-16e294018ccf" containerID="1962f31cdcbb3fc9bc6ec7676c633c70bbea935bce89a263ea07d86062101529" exitCode=1 Oct 30 21:47:00.970579 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.970560 5607 manager.go:324] Recovery completed Oct 30 21:47:00.991322 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.991289 5607 generic.go:356] "Generic (PLEG): container finished" podID="f4bae0c6b7f3f9e1322bca7f9070d553" containerID="17771905f9885883d90f69d83a3f0a9e3bd4f5d0e61bd7ebb6e88de6c916a550" exitCode=0 Oct 30 21:47:00.999549 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:00.999518 5607 generic.go:356] "Generic (PLEG): container finished" podID="ab980c40-b10c-4f42-90d8-bd930becdb72" containerID="c6828a656afe137548eb5bc42fd653c14a918df3ef8e15be60b45a6cfeb98013" exitCode=0 Oct 30 21:47:01.003630 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.002484 5607 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:47:01.009673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009651 5607 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:47:01.009673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009670 5607 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:47:01.009824 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009691 5607 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:47:01.009874 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009858 5607 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:47:01.009934 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009874 5607 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:47:01.009934 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009897 5607 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:47:01.010050 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.009936 5607 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:47:01.010521 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.010504 5607 policy_none.go:50] "Start" Oct 30 21:47:01.010521 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.010525 5607 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:47:01.010656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.010541 5607 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:47:01.010703 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.010698 5607 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:47:01.010745 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.010707 5607 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:47:01.010745 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.010713 5607 policy_none.go:44] "Start" Oct 30 21:47:01.017079 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.016437 5607 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:47:01.024099 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024083 5607 manager.go:341] "Starting Device Plugin manager" Oct 30 21:47:01.024195 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:01.024115 5607 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:47:01.024195 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024128 5607 server.go:87] "Starting device plugin registration server" Oct 30 21:47:01.024492 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024476 5607 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:47:01.024566 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024494 5607 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:47:01.024764 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024743 5607 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:47:01.024882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024839 5607 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:47:01.024882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024880 5607 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:47:01.025012 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024888 5607 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:47:01.025064 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.024986 5607 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:47:01.024869564Z","UUID":"1232fece-6824-4100-b442-a512eb88b4db","Handler":null,"Name":"","Endpoint":""} Oct 30 21:47:01.026713 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:01.026690 5607 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:47:01.027332 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.027308 5607 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:47:01.027431 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.027338 5607 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:47:01.046055 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.046036 5607 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:47:01.125320 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.125288 5607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:47:01.127363 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.127339 5607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:47:01.127483 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.127375 5607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:47:01.127483 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.127388 5607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-91.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:47:01.127483 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.127448 5607 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.162862 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.162802 5607 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal"] Oct 30 21:47:01.163622 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163596 5607 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.163732 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163629 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"4385e5f7b5de508595fd215a6736486a","Type":"ContainerStarted","Data":"8fc10c207de2af15c1ac10e1b1c1d1881f18ff8b476005ec6f2ed0ffa163353b"} Oct 30 21:47:01.163732 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163684 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"4385e5f7b5de508595fd215a6736486a","Type":"ContainerStarted","Data":"2fc725da30a3d609b419f48956ca12bb86433848cfcdae4315ad4a34fc3b6103"} Oct 30 21:47:01.163848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163736 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"f4bae0c6b7f3f9e1322bca7f9070d553","Type":"ContainerStarted","Data":"b45955a0a68b04a68b82533c8153db49d015910665e6ed077cd7c23da30e777c"} Oct 30 21:47:01.163848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163757 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"f4bae0c6b7f3f9e1322bca7f9070d553","Type":"ContainerDied","Data":"17771905f9885883d90f69d83a3f0a9e3bd4f5d0e61bd7ebb6e88de6c916a550"} Oct 30 21:47:01.163848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163775 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" event={"ID":"f4bae0c6b7f3f9e1322bca7f9070d553","Type":"ContainerStarted","Data":"5062cb9d6966ebfec5936414447787c0082965d1e8289a895de356aa0d877aab"} Oct 30 21:47:01.163848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.163820 5607 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.221989 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.221964 5607 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:47:01.222130 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:01.222060 5607 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.226311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.226274 5607 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:47:01.226432 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:01.226325 5607 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.231053 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.231030 5607 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.231150 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.231096 5607 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.258588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.258534 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4385e5f7b5de508595fd215a6736486a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"4385e5f7b5de508595fd215a6736486a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.258588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.258561 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.258588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.258586 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.359411 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.359378 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4385e5f7b5de508595fd215a6736486a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"4385e5f7b5de508595fd215a6736486a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.359651 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.359620 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.359765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.359656 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.359765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.359500 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4385e5f7b5de508595fd215a6736486a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"4385e5f7b5de508595fd215a6736486a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.359765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.359708 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.359765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.359726 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4bae0c6b7f3f9e1322bca7f9070d553-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" (UID: \"f4bae0c6b7f3f9e1322bca7f9070d553\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:01.753040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:01.753008 5607 apiserver.go:52] "Watching apiserver" Oct 30 21:47:02.013413 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.013342 5607 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:02.013562 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.013492 5607 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:02.025858 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.025830 5607 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:47:02.028476 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.027680 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-687l6","openshift-multus/network-metrics-daemon-7lq8c","openshift-cluster-node-tuning-operator/tuned-pj2jw","openshift-multus/multus-bvk6g","openshift-dns/dns-default-cjx5v","openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh","openshift-console/console-544774fdb7-dzpt4","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-zbzd4","openshift-ovn-kubernetes/ovnkube-node-2s925","kube-system/konnectivity-agent-dql2n","openshift-monitoring/node-exporter-fq2xz","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4","openshift-console/console-664c5b7ff9-c52hh","openshift-image-registry/node-ca-t8l67","openshift-insights/insights-runtime-extractor-phd9k","openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf","openshift-multus/multus-additional-cni-plugins-8xjbc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-gqkkp","openshift-dns/node-resolver-92scn","openshift-monitoring/thanos-querier-6f9b796bff-x4mrn","openshift-console/downloads-6dd789c8b7-jl9q2","openshift-ingress-canary/ingress-canary-4f8cw","kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal","openshift-image-registry/image-registry-c78b64cb4-4q2c8"] Oct 30 21:47:02.057981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.057940 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.071243 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.071223 5607 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:47:02.071343 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:02.071264 5607 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:02.072017 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.071998 5607 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:47:02.072117 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:02.072041 5607 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.us-west-2.compute.internal" Oct 30 21:47:02.079231 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.079167 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.079321 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.079252 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.079321 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.079304 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.079423 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.079380 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.080360 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.080341 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:47:02.080500 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.080398 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.080500 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.080408 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.081099 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081078 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.081194 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081159 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.081194 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081188 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:47:02.081300 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081259 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.081345 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081334 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.081446 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081423 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:47:02.081546 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081489 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.081595 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.081588 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.096300 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.096277 5607 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:47:02.111466 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111439 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:47:02.111559 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111529 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:47:02.111631 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111587 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:47:02.111631 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111626 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.111740 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111683 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.111893 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111845 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.111893 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111854 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:47:02.112138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111927 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:47:02.112138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.111983 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:47:02.112138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112012 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:47:02.112138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112065 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:47:02.112138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112127 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.112526 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112219 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:47:02.112526 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112458 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.112526 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112502 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:47:02.112676 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112574 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.112676 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112582 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.112759 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112693 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.112885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112839 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:47:02.112942 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112928 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.112942 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.112938 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:47:02.113179 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.113154 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:47:02.113385 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.113368 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:47:02.113561 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.113540 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.125172 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.125149 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:47:02.125275 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.125179 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:47:02.125275 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.125223 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:47:02.136851 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.136828 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:47:02.137089 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.137073 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:47:02.138571 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.138548 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:47:02.138571 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.138556 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:47:02.159186 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159162 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:47:02.159282 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159192 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mkvf\"" Oct 30 21:47:02.159282 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159197 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4gblc\"" Oct 30 21:47:02.159282 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159197 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:47:02.159437 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159299 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:47:02.159500 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159481 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g8m85\"" Oct 30 21:47:02.159550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159506 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159618 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159729 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kjn6x\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159783 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159790 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-879p5\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159791 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159820 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a50jmvgl924dq\"" Oct 30 21:47:02.159848 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159839 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159859 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159870 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159889 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bg2hw\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159889 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159935 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159969 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.159882 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160194 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160203 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:47:02.160222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160208 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mpsf4\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160239 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160263 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qq6nx\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160287 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160300 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160317 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hlprr\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160334 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pz9m9\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160355 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fd5qx\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160366 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160407 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5wwct\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160459 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vvlch\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160494 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvjtk\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160507 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160546 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160546 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rn9hr\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160551 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160585 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160594 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:47:02.160672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160645 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f29teacdhvtdb\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160691 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9dfpb\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160694 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160696 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hs26f\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160744 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kp2zt\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160818 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cpxgn\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160830 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160860 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160901 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.160943 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.161089 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.161161 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-98lqs\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.161221 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqqg2\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.161348 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1papng1cqcjqm\"" Oct 30 21:47:02.161467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.161459 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:47:02.162020 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.161574 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:47:02.165168 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165141 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.165281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165182 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-kubelet\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.165281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165210 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-hostroot\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.165281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165232 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-run\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.165281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165267 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6bl\" (UniqueName: \"kubernetes.io/projected/3ebfccd4-cbd5-4319-be94-3da3df1d8566-kube-api-access-7k6bl\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.165472 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165316 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jht\" (UniqueName: \"kubernetes.io/projected/bb95c8ea-e1a6-40f5-963f-19005925a1f8-kube-api-access-c9jht\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.165528 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165474 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.165642 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165612 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-modprobe-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.165731 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165653 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-conf\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.165731 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165710 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80e5d657-00d6-433e-b9bd-0ca172deb64d-metrics-tls\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.165842 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165741 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-lib-modules\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.165842 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165767 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovn-node-metrics-cert\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.165842 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165791 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-tls\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.165992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165865 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.165992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165886 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-cni-binary-copy\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.165992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165919 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-socket-dir-parent\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.165992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165938 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-dbus\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.165992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165972 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.166203 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.165995 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-env-overrides\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.166203 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166020 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-data-volume\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.166281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166264 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-env-overrides\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.166337 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166324 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.166376 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166348 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcwv\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-kube-api-access-txcwv\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.166425 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166371 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.166425 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166398 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.166425 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166421 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-textfile\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.166547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166443 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.166547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166466 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.166547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166482 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e5d657-00d6-433e-b9bd-0ca172deb64d-tmp-dir\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.166547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166505 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5bq\" (UniqueName: \"kubernetes.io/projected/80e5d657-00d6-433e-b9bd-0ca172deb64d-kube-api-access-gg5bq\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.166547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166532 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhxz\" (UniqueName: \"kubernetes.io/projected/8639240c-cb7f-45b0-a0ce-97eb850716cc-kube-api-access-skhxz\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.166738 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166557 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-bin\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.166738 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166586 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.166738 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166611 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-conf-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.166738 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166634 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-sys\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.166882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166792 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-host\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.166882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166811 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-ovn\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.166882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166827 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.166882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166856 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166889 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-script-lib\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166916 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8dbcc04-78d4-4ac0-8697-deec99270f95-tmp\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166964 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-systemd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.166990 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-grpc-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167014 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167046 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.167076 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167073 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167090 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/285863f3-fa2b-4d9a-80b2-7421899e97c4-serviceca\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167106 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-registration-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167121 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-host-slash\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167140 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-cnibin\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167167 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfccd4-cbd5-4319-be94-3da3df1d8566-metrics-client-ca\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167197 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167220 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-sys\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167243 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-etc-kubernetes\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167270 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-crio-socket\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167292 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.167334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167318 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdxt\" (UniqueName: \"kubernetes.io/projected/a8dbcc04-78d4-4ac0-8697-deec99270f95-kube-api-access-9hdxt\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167342 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167376 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167407 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167431 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe895c1-e716-45d5-94eb-58668c4bd1c1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167453 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-sys-fs\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167780 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe895c1-e716-45d5-94eb-58668c4bd1c1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.167830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167823 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgds\" (UniqueName: \"kubernetes.io/projected/2e9328b7-cd5f-4a63-8895-be053051f188-kube-api-access-nkgds\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167868 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-netns\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167890 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-log-socket\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167920 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167968 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729a10ce-45d9-4094-8f2b-c5af2b19d159-agent-certs\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.167997 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80e5d657-00d6-433e-b9bd-0ca172deb64d-config-volume\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168022 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-system-cni-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168047 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168055 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-data-volume\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168099 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl64\" (UniqueName: \"kubernetes.io/projected/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-kube-api-access-4vl64\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168127 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-root\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.168165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168154 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.168658 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168181 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.168658 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168213 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebfccd4-cbd5-4319-be94-3da3df1d8566-metrics-client-ca\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.168658 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168324 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-tls\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.168658 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168524 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.168658 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168626 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-script-lib\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.168913 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168692 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8dbcc04-78d4-4ac0-8697-deec99270f95-tmp\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:47:02.168913 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168728 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80e5d657-00d6-433e-b9bd-0ca172deb64d-metrics-tls\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.168913 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168857 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.169073 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168935 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-grpc-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.169073 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168945 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovn-node-metrics-cert\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.169180 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169160 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-cni-binary-copy\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.169354 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169247 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.169354 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169377 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.169641 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169484 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.169641 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.168216 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qt92\" (UniqueName: \"kubernetes.io/projected/35cd7059-12c6-4c41-99ef-86eea5a5873e-kube-api-access-8qt92\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169854 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/285863f3-fa2b-4d9a-80b2-7421899e97c4-serviceca\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169884 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169898 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169942 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.169970 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-slash\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170000 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-var-lib-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170028 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-textfile\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170032 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhtv\" (UniqueName: \"kubernetes.io/projected/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-kube-api-access-5dhtv\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170059 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170085 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-kubelet-config\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170092 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e5d657-00d6-433e-b9bd-0ca172deb64d-tmp-dir\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.170111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170111 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-etc-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170147 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-iptables-alerter-script\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170157 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/729a10ce-45d9-4094-8f2b-c5af2b19d159-agent-certs\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170174 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170197 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170222 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-tls\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170331 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80e5d657-00d6-433e-b9bd-0ca172deb64d-config-volume\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170390 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-certificates\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170427 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ebfccd4-cbd5-4319-be94-3da3df1d8566-tmp\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170457 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170486 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-device-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170513 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170540 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-multus\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170543 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-iptables-alerter-script\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170571 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170600 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170601 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170631 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cd9p\" (UniqueName: \"kubernetes.io/projected/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kube-api-access-5cd9p\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170696 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-tls\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.170847 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171006 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe895c1-e716-45d5-94eb-58668c4bd1c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171055 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-client-certs\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171094 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171115 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171178 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-registry-certificates\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171189 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171232 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1bdaec03-c2f8-446e-9376-bed9a55677c9-original-pull-secret\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171239 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ebfccd4-cbd5-4319-be94-3da3df1d8566-tmp\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171325 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-secret-metrics-server-client-certs\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171368 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171410 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-cnibin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171430 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.171525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171458 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-k8s-cni-cncf-io\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171580 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171734 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729a10ce-45d9-4094-8f2b-c5af2b19d159-konnectivity-ca\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171771 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171978 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/729a10ce-45d9-4094-8f2b-c5af2b19d159-konnectivity-ca\") pod \"konnectivity-agent-dql2n\" (UID: \"729a10ce-45d9-4094-8f2b-c5af2b19d159\") " pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.171935 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172178 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-kubernetes\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172217 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ebfccd4-cbd5-4319-be94-3da3df1d8566-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172227 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172290 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172403 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-netns\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172461 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172480 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172520 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-tmp\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172524 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-accelerators-collector-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172578 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172623 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-tmp\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172663 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdsn\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172694 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172723 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172807 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-cert\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:47:02.172939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.172918 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35cd7059-12c6-4c41-99ef-86eea5a5873e-metrics-certs\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173005 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-system-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173016 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173060 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-bin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173102 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173145 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-node-log\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173182 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-netd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173214 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-cert\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173230 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e46f2-7b2a-4518-861c-fb94540b0231-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tgqt4\" (UID: \"6f4e46f2-7b2a-4518-861c-fb94540b0231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173297 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcbwm\" (UniqueName: \"kubernetes.io/projected/ab980c40-b10c-4f42-90d8-bd930becdb72-kube-api-access-vcbwm\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173388 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-multus-certs\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173468 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e46f2-7b2a-4518-861c-fb94540b0231-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tgqt4\" (UID: \"6f4e46f2-7b2a-4518-861c-fb94540b0231\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173553 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkft\" (UniqueName: \"kubernetes.io/projected/ffe895c1-e716-45d5-94eb-58668c4bd1c1-kube-api-access-dqkft\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173594 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-socket-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173667 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8639240c-cb7f-45b0-a0ce-97eb850716cc-hosts-file\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173731 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8639240c-cb7f-45b0-a0ce-97eb850716cc-tmp-dir\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173750 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173786 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-metrics-server-audit-profiles\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.173813 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/285863f3-fa2b-4d9a-80b2-7421899e97c4-host\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.174003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174009 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8639240c-cb7f-45b0-a0ce-97eb850716cc-tmp-dir\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174022 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174072 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-systemd-units\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174109 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-metrics-client-ca\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174127 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174150 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174175 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cw9d\" (UniqueName: \"kubernetes.io/projected/285863f3-fa2b-4d9a-80b2-7421899e97c4-kube-api-access-2cw9d\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174196 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-trusted-ca\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174219 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prh52\" (UniqueName: \"kubernetes.io/projected/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-api-access-prh52\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174244 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb95c8ea-e1a6-40f5-963f-19005925a1f8-audit-log\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174267 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174287 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174301 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab980c40-b10c-4f42-90d8-bd930becdb72-metrics-client-ca\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174312 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174395 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bb95c8ea-e1a6-40f5-963f-19005925a1f8-audit-log\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174566 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.174721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174712 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-image-registry-private-configuration\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174771 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5619b33-0539-4471-ac48-e5f51da5ab38-trusted-ca\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174808 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-var-lib-kubelet\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174919 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-kubelet\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174935 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-image-registry-private-configuration\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174973 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-client-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.174999 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-tls\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175031 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175057 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175190 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-tls\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175217 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-installation-pull-secrets\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175255 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysconfig\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175283 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-systemd\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175321 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.175404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175325 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bb95c8ea-e1a6-40f5-963f-19005925a1f8-client-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175422 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175458 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175485 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5619b33-0539-4471-ac48-e5f51da5ab38-ca-trust-extracted\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175527 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5619b33-0539-4471-ac48-e5f51da5ab38-installation-pull-secrets\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175566 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5619b33-0539-4471-ac48-e5f51da5ab38-ca-trust-extracted\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175599 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-wtmp\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175633 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175645 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175656 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175663 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175693 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175822 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175849 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.175981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.175942 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-multus-daemon-config\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176007 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kb5\" (UniqueName: \"kubernetes.io/projected/191e72fd-de5a-4dea-9592-c75367da8608-kube-api-access-55kb5\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176040 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-bound-sa-token\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176068 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-config\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176089 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176105 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176156 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176227 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-etc-tuned\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176246 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/191e72fd-de5a-4dea-9592-c75367da8608-multus-daemon-config\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176257 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75sw\" (UniqueName: \"kubernetes.io/projected/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-kube-api-access-k75sw\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176316 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-ovnkube-config\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176333 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e9328b7-cd5f-4a63-8895-be053051f188-etc-tuned\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176418 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/248fb46a-f3b1-4d89-9d88-0191280d983d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176467 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsts\" (UniqueName: \"kubernetes.io/projected/248fb46a-f3b1-4d89-9d88-0191280d983d-kube-api-access-pjsts\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.176621 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.176471 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.179084 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.179063 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:47:02.179084 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.179073 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:47:02.179230 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.179096 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:47:02.179230 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.179124 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:47:02.179230 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.179168 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:47:02.181746 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.181686 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.185696 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.185000 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bb95c8ea-e1a6-40f5-963f-19005925a1f8-metrics-server-audit-profiles\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.205568 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.205520 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:47:02.205701 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.205676 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:47:02.205824 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.205801 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:47:02.211630 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.211610 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.221914 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.221888 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6bl\" (UniqueName: \"kubernetes.io/projected/3ebfccd4-cbd5-4319-be94-3da3df1d8566-kube-api-access-7k6bl\") pod \"thanos-querier-6f9b796bff-x4mrn\" (UID: \"3ebfccd4-cbd5-4319-be94-3da3df1d8566\") " pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:02.222470 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.222447 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jht\" (UniqueName: \"kubernetes.io/projected/bb95c8ea-e1a6-40f5-963f-19005925a1f8-kube-api-access-c9jht\") pod \"metrics-server-5fd6cdc7bc-bdwzf\" (UID: \"bb95c8ea-e1a6-40f5-963f-19005925a1f8\") " pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:02.224525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.224503 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qt92\" (UniqueName: \"kubernetes.io/projected/35cd7059-12c6-4c41-99ef-86eea5a5873e-kube-api-access-8qt92\") pod \"network-metrics-daemon-7lq8c\" (UID: \"35cd7059-12c6-4c41-99ef-86eea5a5873e\") " pod="openshift-multus/network-metrics-daemon-7lq8c" Oct 30 21:47:02.225447 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.225428 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") pod \"console-544774fdb7-dzpt4\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:02.225878 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.225859 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxksq\" (UniqueName: \"kubernetes.io/projected/90fc0a23-c03b-40f3-83ac-61ba7b26bc5e-kube-api-access-hxksq\") pod \"network-check-target-zbzd4\" (UID: \"90fc0a23-c03b-40f3-83ac-61ba7b26bc5e\") " pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:47:02.226145 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.226120 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgds\" (UniqueName: \"kubernetes.io/projected/2e9328b7-cd5f-4a63-8895-be053051f188-kube-api-access-nkgds\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.226293 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.226267 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdxt\" (UniqueName: \"kubernetes.io/projected/a8dbcc04-78d4-4ac0-8697-deec99270f95-kube-api-access-9hdxt\") pod \"downloads-6dd789c8b7-jl9q2\" (UID: \"a8dbcc04-78d4-4ac0-8697-deec99270f95\") " pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:47:02.226381 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.226304 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl64\" (UniqueName: \"kubernetes.io/projected/0140305c-d4d1-4ae9-877a-6c6d21b80d0d-kube-api-access-4vl64\") pod \"ingress-canary-4f8cw\" (UID: \"0140305c-d4d1-4ae9-877a-6c6d21b80d0d\") " pod="openshift-ingress-canary/ingress-canary-4f8cw" Oct 30 21:47:02.227496 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.227131 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhxz\" (UniqueName: \"kubernetes.io/projected/8639240c-cb7f-45b0-a0ce-97eb850716cc-kube-api-access-skhxz\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.227496 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.227304 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhtv\" (UniqueName: \"kubernetes.io/projected/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-kube-api-access-5dhtv\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.227496 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.227469 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcwv\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-kube-api-access-txcwv\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.230186 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.230163 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") pod \"console-664c5b7ff9-c52hh\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:02.231834 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.231812 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5bq\" (UniqueName: \"kubernetes.io/projected/80e5d657-00d6-433e-b9bd-0ca172deb64d-kube-api-access-gg5bq\") pod \"dns-default-cjx5v\" (UID: \"80e5d657-00d6-433e-b9bd-0ca172deb64d\") " pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:02.231910 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.231874 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcbwm\" (UniqueName: \"kubernetes.io/projected/ab980c40-b10c-4f42-90d8-bd930becdb72-kube-api-access-vcbwm\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.232382 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.232364 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cw9d\" (UniqueName: \"kubernetes.io/projected/285863f3-fa2b-4d9a-80b2-7421899e97c4-kube-api-access-2cw9d\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.232515 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.232494 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkft\" (UniqueName: \"kubernetes.io/projected/ffe895c1-e716-45d5-94eb-58668c4bd1c1-kube-api-access-dqkft\") pod \"openshift-state-metrics-7bd7489957-9gxsh\" (UID: \"ffe895c1-e716-45d5-94eb-58668c4bd1c1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-9gxsh" Oct 30 21:47:02.232827 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.232808 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cd9p\" (UniqueName: \"kubernetes.io/projected/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kube-api-access-5cd9p\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.242329 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.242300 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75sw\" (UniqueName: \"kubernetes.io/projected/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-kube-api-access-k75sw\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.243440 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.243417 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5619b33-0539-4471-ac48-e5f51da5ab38-bound-sa-token\") pod \"image-registry-c78b64cb4-4q2c8\" (UID: \"b5619b33-0539-4471-ac48-e5f51da5ab38\") " pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:02.243532 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.243496 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prh52\" (UniqueName: \"kubernetes.io/projected/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-kube-api-access-prh52\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.243651 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.243631 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kb5\" (UniqueName: \"kubernetes.io/projected/191e72fd-de5a-4dea-9592-c75367da8608-kube-api-access-55kb5\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.248470 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.248448 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsts\" (UniqueName: \"kubernetes.io/projected/248fb46a-f3b1-4d89-9d88-0191280d983d-kube-api-access-pjsts\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.277594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277539 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-wtmp\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.277594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277577 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.277742 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277603 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.277742 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277620 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-node-exporter-wtmp\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.277742 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277632 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.277742 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277705 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.277742 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277709 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-kubelet\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277744 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-hostroot\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277754 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-kubelet\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277767 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-run\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277794 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-modprobe-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277801 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-hostroot\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277769 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277816 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-conf\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277840 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-lib-modules\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277866 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277892 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-socket-dir-parent\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.277916 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-dbus\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.278019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278003 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-dbus\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278029 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-lib-modules\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278057 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-socket-dir-parent\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278079 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-conf\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278124 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278159 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278186 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278216 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-bin\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278240 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-conf-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278260 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-sys\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278269 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-modprobe-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278282 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-host\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278296 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysctl-d\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278320 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-ovn\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278330 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-host\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278351 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278372 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-multus-conf-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278380 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278409 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-systemd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278416 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-sys\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278351 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-bin\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278435 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278475 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-registration-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278501 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-host-slash\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278525 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-cnibin\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278552 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-sys\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.278557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278575 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-etc-kubernetes\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278600 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-crio-socket\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278610 5607 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278625 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278648 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278674 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-sys-fs\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278698 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-netns\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278728 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-run\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278750 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-log-socket\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278778 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278809 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-system-cni-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278835 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278880 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-systemd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278896 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278935 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-root\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.278967 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-registration-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279002 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279008 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-root\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279031 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-slash\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279064 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-var-lib-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279089 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279108 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-kubelet-config\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279136 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-etc-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279194 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-etc-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279198 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-ovn\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279234 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-sys-fs\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279247 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-device-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279263 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-netns\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279273 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279294 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-log-socket\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279300 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-multus\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279330 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279334 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279355 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-cnibin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279375 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-system-cni-dir\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279381 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-k8s-cni-cncf-io\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279422 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-k8s-cni-cncf-io\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279430 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279464 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-kubernetes\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279471 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb385ed2-d5f0-4025-b614-1db5e6f0dd68-host-slash\") pod \"iptables-alerter-gqkkp\" (UID: \"cb385ed2-d5f0-4025-b614-1db5e6f0dd68\") " pod="openshift-network-operator/iptables-alerter-gqkkp" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279474 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-device-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279499 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-netns\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279522 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/248fb46a-f3b1-4d89-9d88-0191280d983d-cnibin\") pod \"multus-additional-cni-plugins-8xjbc\" (UID: \"248fb46a-f3b1-4d89-9d88-0191280d983d\") " pod="openshift-multus/multus-additional-cni-plugins-8xjbc" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279527 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279554 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279574 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab980c40-b10c-4f42-90d8-bd930becdb72-sys\") pod \"node-exporter-fq2xz\" (UID: \"ab980c40-b10c-4f42-90d8-bd930becdb72\") " pod="openshift-monitoring/node-exporter-fq2xz" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279581 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdsn\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279609 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-system-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279634 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-bin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279658 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279682 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-node-log\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279710 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-netd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279737 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-multus-certs\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279762 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-socket-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279610 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-etc-kubernetes\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279786 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8639240c-cb7f-45b0-a0ce-97eb850716cc-hosts-file\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279828 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd3fa78-0fa0-41de-b566-bc0cea25b5b4-crio-socket\") pod \"insights-runtime-extractor-phd9k\" (UID: \"8fd3fa78-0fa0-41de-b566-bc0cea25b5b4\") " pod="openshift-insights/insights-runtime-extractor-phd9k" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279852 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/285863f3-fa2b-4d9a-80b2-7421899e97c4-host\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279878 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-systemd-units\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279906 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279932 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.279995 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280020 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.280009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280045 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-var-lib-kubelet\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280069 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-kubelet\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280100 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280126 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysconfig\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280149 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-systemd\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.280264 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-systemd\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281155 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281287 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-multus\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281413 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281492 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-cnibin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281542 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-node-log\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281582 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-slash\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281623 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-var-lib-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281902 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.281986 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1bdaec03-c2f8-446e-9376-bed9a55677c9-kubelet-config\") pod \"global-pull-secret-syncer-687l6\" (UID: \"1bdaec03-c2f8-446e-9376-bed9a55677c9\") " pod="kube-system/global-pull-secret-syncer-687l6" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.282027 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.282076 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-kubernetes\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.282113 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-netns\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.282155 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.283506 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-system-cni-dir\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.283560 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-var-lib-cni-bin\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.283604 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-run-openvswitch\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.285669 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285253 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.285669 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285439 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.285669 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285521 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-var-lib-kubelet\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.285825 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285676 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.285825 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285756 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a-socket-dir\") pod \"aws-ebs-csi-driver-node-w4f9n\" (UID: \"f1ba5b0e-7fee-4e39-a4bf-a6ac1e9c488a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4f9n" Oct 30 21:47:02.285825 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285803 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-cni-netd\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.285992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.285846 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/191e72fd-de5a-4dea-9592-c75367da8608-host-run-multus-certs\") pod \"multus-bvk6g\" (UID: \"191e72fd-de5a-4dea-9592-c75367da8608\") " pod="openshift-multus/multus-bvk6g" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286208 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286282 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/285863f3-fa2b-4d9a-80b2-7421899e97c4-host\") pod \"node-ca-t8l67\" (UID: \"285863f3-fa2b-4d9a-80b2-7421899e97c4\") " pod="openshift-image-registry/node-ca-t8l67" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286498 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8639240c-cb7f-45b0-a0ce-97eb850716cc-hosts-file\") pod \"node-resolver-92scn\" (UID: \"8639240c-cb7f-45b0-a0ce-97eb850716cc\") " pod="openshift-dns/node-resolver-92scn" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286544 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-systemd-units\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286600 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e9328b7-cd5f-4a63-8895-be053051f188-etc-sysconfig\") pod \"tuned-pj2jw\" (UID: \"2e9328b7-cd5f-4a63-8895-be053051f188\") " pod="openshift-cluster-node-tuning-operator/tuned-pj2jw" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286661 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cf0a1a-efb8-4aef-ac2a-16e294018ccf-host-kubelet\") pod \"ovnkube-node-2s925\" (UID: \"18cf0a1a-efb8-4aef-ac2a-16e294018ccf\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286771 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.287885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.286928 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.288324 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.288067 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.289351 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.288862 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.289351 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.289061 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.289705 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.289677 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.290275 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.290253 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.291086 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.291065 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.291330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.291311 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.291408 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.291379 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.292704 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.292683 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.293380 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.293362 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.342063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.342026 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdsn\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn\") pod \"prometheus-k8s-0\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.343283 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.343264 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:47:02.517726 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.517692 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:02.642027 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.641543 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zbzd4" Oct 30 21:47:02.745719 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:02.745681 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c78b64cb4-4q2c8" Oct 30 21:47:03.106134 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:03.106104 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:03.111008 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:03.110932 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64ca20a_ac6f_4eeb_8f95_9a7cc8b35cd8.slice/crio-0ab95e44cbbb931685da1667a5e8dbcb6877f0b7cae2c7da4df9cca2cabb3fce WatchSource:0}: Error finding container 0ab95e44cbbb931685da1667a5e8dbcb6877f0b7cae2c7da4df9cca2cabb3fce: Status 404 returned error can't find the container with id 0ab95e44cbbb931685da1667a5e8dbcb6877f0b7cae2c7da4df9cca2cabb3fce Oct 30 21:47:03.683775 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:03.683743 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-544774fdb7-dzpt4"] Oct 30 21:47:04.025612 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:04.025582 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"f7b44c224414de35bce2e444b9a4aef057ab47bcf6c961aa49553f468bc2ec1e"} Oct 30 21:47:04.027166 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:04.027097 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"0ab95e44cbbb931685da1667a5e8dbcb6877f0b7cae2c7da4df9cca2cabb3fce"} Oct 30 21:47:04.029001 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:04.028940 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" event={"ID":"bb95c8ea-e1a6-40f5-963f-19005925a1f8","Type":"ContainerStarted","Data":"639b398e130e88f572aeac1e2a19ff954e4235027d51467c644257bd3ebafbd8"} Oct 30 21:47:04.693327 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:04.693271 5607 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dql2n_729a10ce-45d9-4094-8f2b-c5af2b19d159/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:04.920737 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:04.920703 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cjx5v" Oct 30 21:47:05.037281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:05.037195 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"7011e3cb9fc998bc9569bfecbef95746acd3f2146924c8764b5f411931b30854"} Oct 30 21:47:05.037453 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:05.037314 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" event={"ID":"3ebfccd4-cbd5-4319-be94-3da3df1d8566","Type":"ContainerStarted","Data":"340849f9669bed999a72f7c866913302a0f1ad9f1cbb9c303cf001994b51c6c0"} Oct 30 21:47:05.987088 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:05.987055 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:06.043488 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.043454 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84" exitCode=0 Oct 30 21:47:06.043648 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.043607 5607 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:47:06.044117 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.044088 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84"} Oct 30 21:47:06.349656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.349581 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:06.504827 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.504790 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:06.669660 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.669513 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tgqt4" Oct 30 21:47:06.701159 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.701135 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:06.715021 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.714160 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:47:06.717302 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.717277 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-jl9q2" Oct 30 21:47:06.901317 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.901288 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:06.907684 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:06.907660 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f9b796bff-x4mrn" Oct 30 21:47:07.620704 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:07.620672 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:07.976085 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:07.975655 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:07.976085 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:07.975886 5607 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:47:08.470223 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:08.470194 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dql2n" Oct 30 21:47:11.067675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.067638 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251"} Oct 30 21:47:11.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.067716 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a"} Oct 30 21:47:11.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.067760 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf"} Oct 30 21:47:11.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.067797 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba"} Oct 30 21:47:11.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.067840 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca"} Oct 30 21:47:11.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.067875 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerStarted","Data":"f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462"} Oct 30 21:47:11.206934 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.206909 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:11.297093 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:11.297026 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.6120854 podStartE2EDuration="10.297013022s" totalImagesPullingTime="6.662222327s" totalInitContainerRuntime="22.705295ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:11.250280439 +0000 UTC m=+10.650836560" watchObservedRunningTime="2026-10-30 21:47:11.297013022 +0000 UTC m=+10.697569143" Oct 30 21:47:14.638300 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:14.638268 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s925" Oct 30 21:47:15.991944 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:15.991913 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:15.996667 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:15.996643 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:47:19.595325 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:19.595299 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:21.220251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:21.220215 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:21.263749 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:21.263726 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:22.128141 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:22.128115 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:22.863517 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:22.863481 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:47:28.712555 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:28.712510 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-544774fdb7-dzpt4" podUID="8cb23260-4a0e-44e6-8023-124e58c32c5e" containerName="console" containerID="cri-o://23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44" gracePeriod=15 Oct 30 21:47:28.989338 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:28.989282 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:29.098539 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098513 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098551 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098597 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098628 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098805 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098675 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098805 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098722 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098805 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098776 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") pod \"8cb23260-4a0e-44e6-8023-124e58c32c5e\" (UID: \"8cb23260-4a0e-44e6-8023-124e58c32c5e\") " Oct 30 21:47:29.098984 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.098911 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:29.099060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.099036 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:29.099115 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.099100 5607 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cb23260-4a0e-44e6-8023-124e58c32c5e-tmp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.099249 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.099227 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:29.099375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.099356 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:29.100967 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.100926 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:29.101066 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.101037 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:29.101110 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.101098 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw" pod "8cb23260-4a0e-44e6-8023-124e58c32c5e" (UID: "8cb23260-4a0e-44e6-8023-124e58c32c5e"). InnerVolumeSpecName "kube-api-access-wd9xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:29.134413 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.134385 5607 generic.go:356] "Generic (PLEG): container finished" podID="8cb23260-4a0e-44e6-8023-124e58c32c5e" containerID="23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44" exitCode=0 Oct 30 21:47:29.134502 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.134427 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544774fdb7-dzpt4" Oct 30 21:47:29.134542 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.134429 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544774fdb7-dzpt4" event={"ID":"8cb23260-4a0e-44e6-8023-124e58c32c5e","Type":"ContainerDied","Data":"23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44"} Oct 30 21:47:29.134576 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.134550 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544774fdb7-dzpt4" event={"ID":"8cb23260-4a0e-44e6-8023-124e58c32c5e","Type":"ContainerDied","Data":"8486dbc43f8a2e4253a991723d16b063e060d7057cd8ee2d6bed3308b57922f2"} Oct 30 21:47:29.134613 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.134576 5607 scope.go:122] "RemoveContainer" containerID="23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44" Oct 30 21:47:29.143636 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.143616 5607 scope.go:122] "RemoveContainer" containerID="23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44" Oct 30 21:47:29.143893 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:29.143875 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44\": container with ID starting with 23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44 not found: ID does not exist" containerID="23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44" Oct 30 21:47:29.143983 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.143905 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44"} err="failed to get container status \"23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44\": rpc error: code = NotFound desc = could not find container \"23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44\": container with ID starting with 23a6d07ec11c06e855c2a99c866c28b5c2a711aef38cbad11fe89041cafd2d44 not found: ID does not exist" Oct 30 21:47:29.200434 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.200410 5607 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-oauth-serving-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.200434 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.200432 5607 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.200550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.200441 5607 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-config\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.200550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.200450 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9xw\" (UniqueName: \"kubernetes.io/projected/8cb23260-4a0e-44e6-8023-124e58c32c5e-kube-api-access-wd9xw\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.200550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.200459 5607 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-serving-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.200550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.200468 5607 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cb23260-4a0e-44e6-8023-124e58c32c5e-console-oauth-config\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:29.293040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.292990 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-544774fdb7-dzpt4"] Oct 30 21:47:29.300940 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:29.300917 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-544774fdb7-dzpt4"] Oct 30 21:47:30.870253 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:30.870216 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb23260-4a0e-44e6-8023-124e58c32c5e" path="/var/lib/kubelet/pods/8cb23260-4a0e-44e6-8023-124e58c32c5e/volumes" Oct 30 21:47:39.601399 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:39.601368 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:39.605143 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:39.605119 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fd6cdc7bc-bdwzf" Oct 30 21:47:52.864009 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:52.863946 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:47:53.633668 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.633633 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:53.634406 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.634366 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="prometheus" containerID="cri-o://f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462" gracePeriod=600 Oct 30 21:47:53.634678 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.634402 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy" containerID="cri-o://9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a" gracePeriod=600 Oct 30 21:47:53.634799 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.634428 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="thanos-sidecar" containerID="cri-o://8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba" gracePeriod=600 Oct 30 21:47:53.634883 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.634486 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251" gracePeriod=600 Oct 30 21:47:53.635849 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.634464 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-web" containerID="cri-o://9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf" gracePeriod=600 Oct 30 21:47:53.635849 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:53.634488 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="config-reloader" containerID="cri-o://c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca" gracePeriod=600 Oct 30 21:47:54.239014 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.238979 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251" exitCode=0 Oct 30 21:47:54.239014 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239007 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a" exitCode=0 Oct 30 21:47:54.239014 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239014 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba" exitCode=0 Oct 30 21:47:54.239014 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239021 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca" exitCode=0 Oct 30 21:47:54.239448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239029 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462" exitCode=0 Oct 30 21:47:54.239448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239048 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251"} Oct 30 21:47:54.239448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239124 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a"} Oct 30 21:47:54.239448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239152 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba"} Oct 30 21:47:54.239448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239187 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca"} Oct 30 21:47:54.239448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.239212 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462"} Oct 30 21:47:54.877840 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.877818 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:54.990078 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990027 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990078 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990058 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990078 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990078 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990283 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990110 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990283 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990140 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990283 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990170 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990283 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990205 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990283 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990234 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990628 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990540 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.990735 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990662 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.990798 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990751 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990798 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990758 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.990905 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990803 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990905 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990824 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990905 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990858 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.990905 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990899 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990937 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.990993 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991038 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991071 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991114 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991147 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991187 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\" (UID: \"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8\") " Oct 30 21:47:54.991676 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991430 5607 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.991676 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991448 5607 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.991676 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.991464 5607 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.992034 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.992006 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.992168 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.992148 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:54.992943 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.992851 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.993223 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.993102 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.993596 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.993567 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.993931 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.993891 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.994130 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.994109 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.994566 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.994539 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:54.994566 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.994554 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.994976 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.994930 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.995455 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.995431 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "kube-api-access-pbdsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:54.995618 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.995593 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.995696 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.995673 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.995876 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.995859 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:54.996563 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.996544 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.996637 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:54.996618 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:55.006459 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.006439 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config" pod "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" (UID: "c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:55.092377 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092359 5607 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-db\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092377 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092378 5607 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092390 5607 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-metrics-client-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092400 5607 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092409 5607 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092418 5607 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092427 5607 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092455 5607 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-web-config\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092464 5607 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-grpc-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092472 5607 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092480 5607 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-metrics-client-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092489 5607 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092498 5607 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092507 5607 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-tls-assets\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092514 5607 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-config-out\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092522 5607 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-secret-kube-rbac-proxy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.092803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.092531 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbdsn\" (UniqueName: \"kubernetes.io/projected/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8-kube-api-access-pbdsn\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:55.245203 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.245131 5607 generic.go:356] "Generic (PLEG): container finished" podID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerID="9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf" exitCode=0 Oct 30 21:47:55.245203 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.245164 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf"} Oct 30 21:47:55.245588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.245221 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8","Type":"ContainerDied","Data":"0ab95e44cbbb931685da1667a5e8dbcb6877f0b7cae2c7da4df9cca2cabb3fce"} Oct 30 21:47:55.245588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.245253 5607 scope.go:122] "RemoveContainer" containerID="5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251" Oct 30 21:47:55.245588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.245270 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.253494 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.253477 5607 scope.go:122] "RemoveContainer" containerID="9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a" Oct 30 21:47:55.261011 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.260993 5607 scope.go:122] "RemoveContainer" containerID="9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf" Oct 30 21:47:55.267405 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.267387 5607 scope.go:122] "RemoveContainer" containerID="8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba" Oct 30 21:47:55.273927 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.273912 5607 scope.go:122] "RemoveContainer" containerID="c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca" Oct 30 21:47:55.280168 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.280153 5607 scope.go:122] "RemoveContainer" containerID="f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462" Oct 30 21:47:55.286808 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.286793 5607 scope.go:122] "RemoveContainer" containerID="f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84" Oct 30 21:47:55.292996 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.292980 5607 scope.go:122] "RemoveContainer" containerID="5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251" Oct 30 21:47:55.293269 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.293238 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251\": container with ID starting with 5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251 not found: ID does not exist" containerID="5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251" Oct 30 21:47:55.293351 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.293262 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251"} err="failed to get container status \"5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251\": rpc error: code = NotFound desc = could not find container \"5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251\": container with ID starting with 5f5e25dc116574033a28316f446188d3d5a5216d5efc7ceb5b808ae028e94251 not found: ID does not exist" Oct 30 21:47:55.293351 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.293284 5607 scope.go:122] "RemoveContainer" containerID="9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a" Oct 30 21:47:55.293538 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.293518 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a\": container with ID starting with 9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a not found: ID does not exist" containerID="9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a" Oct 30 21:47:55.293595 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.293542 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a"} err="failed to get container status \"9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a\": rpc error: code = NotFound desc = could not find container \"9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a\": container with ID starting with 9f3bba8a064298b1d86ac296f6238449461dfc2f0a2f42d85412a63aa9e50d1a not found: ID does not exist" Oct 30 21:47:55.293595 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.293556 5607 scope.go:122] "RemoveContainer" containerID="9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf" Oct 30 21:47:55.293798 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.293780 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf\": container with ID starting with 9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf not found: ID does not exist" containerID="9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf" Oct 30 21:47:55.293889 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.293808 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf"} err="failed to get container status \"9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf\": rpc error: code = NotFound desc = could not find container \"9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf\": container with ID starting with 9b9f5729dea0082234716518437fdd68cc4be69d7293742d292dd61b6253c4bf not found: ID does not exist" Oct 30 21:47:55.293889 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.293830 5607 scope.go:122] "RemoveContainer" containerID="8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba" Oct 30 21:47:55.294076 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.294058 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba\": container with ID starting with 8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba not found: ID does not exist" containerID="8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba" Oct 30 21:47:55.294112 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294083 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba"} err="failed to get container status \"8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba\": rpc error: code = NotFound desc = could not find container \"8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba\": container with ID starting with 8eae76446bdbb64dd1733c00e773a5a43f0dcb28cb98f2b5b500995f642688ba not found: ID does not exist" Oct 30 21:47:55.294112 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294096 5607 scope.go:122] "RemoveContainer" containerID="c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca" Oct 30 21:47:55.294308 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.294292 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca\": container with ID starting with c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca not found: ID does not exist" containerID="c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca" Oct 30 21:47:55.294351 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294316 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca"} err="failed to get container status \"c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca\": rpc error: code = NotFound desc = could not find container \"c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca\": container with ID starting with c01163cd4a470d05e5c52203d5f7a0e1c88b6427a12adbe885ebe71e6f25a4ca not found: ID does not exist" Oct 30 21:47:55.294351 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294335 5607 scope.go:122] "RemoveContainer" containerID="f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462" Oct 30 21:47:55.294544 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.294528 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462\": container with ID starting with f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462 not found: ID does not exist" containerID="f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462" Oct 30 21:47:55.294600 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294552 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462"} err="failed to get container status \"f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462\": rpc error: code = NotFound desc = could not find container \"f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462\": container with ID starting with f71c1de54be91847360ab5df07f50e7b548dca676925801172872dc6f6110462 not found: ID does not exist" Oct 30 21:47:55.294600 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294573 5607 scope.go:122] "RemoveContainer" containerID="f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84" Oct 30 21:47:55.294778 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:47:55.294764 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84\": container with ID starting with f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84 not found: ID does not exist" containerID="f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84" Oct 30 21:47:55.294817 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.294782 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84"} err="failed to get container status \"f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84\": rpc error: code = NotFound desc = could not find container \"f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84\": container with ID starting with f52a1261321492ca7d8f0a9d2b1f200049c6c1bd3e72fbc79cf56cd6a04b3c84 not found: ID does not exist" Oct 30 21:47:55.403859 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.403836 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:55.410190 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.410165 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:55.452590 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452546 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:55.452909 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452882 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="config-reloader" Oct 30 21:47:55.452909 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452907 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="config-reloader" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452920 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="thanos-sidecar" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452925 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="thanos-sidecar" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452932 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="prometheus" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452937 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="prometheus" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452971 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8cb23260-4a0e-44e6-8023-124e58c32c5e" containerName="console" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452977 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb23260-4a0e-44e6-8023-124e58c32c5e" containerName="console" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452986 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-thanos" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.452991 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-thanos" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453005 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453010 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453017 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="init-config-reloader" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453021 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="init-config-reloader" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453027 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-web" Oct 30 21:47:55.453049 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453031 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-web" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453088 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="config-reloader" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453096 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="thanos-sidecar" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453106 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="prometheus" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453114 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb23260-4a0e-44e6-8023-124e58c32c5e" containerName="console" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453120 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-thanos" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453125 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy" Oct 30 21:47:55.453432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.453130 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" containerName="kube-rbac-proxy-web" Oct 30 21:47:55.456693 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.456674 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.508097 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.508042 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:47:55.508097 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.508042 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:47:55.508097 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.508045 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:47:55.509178 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.509141 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:47:55.555087 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555062 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:47:55.555243 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555230 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:47:55.555300 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555249 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:47:55.555300 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555267 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:47:55.555379 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555306 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-98lqs\"" Oct 30 21:47:55.555379 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555339 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f29teacdhvtdb\"" Oct 30 21:47:55.555379 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555346 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:47:55.555379 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555347 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:47:55.555615 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555604 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:47:55.555823 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.555807 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:47:55.556220 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.556202 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:47:55.556309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.556294 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:47:55.557205 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.557191 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:55.596146 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596116 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596252 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596152 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a14e049-63a2-40ed-922d-cf72de38a167-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596252 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596178 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596363 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596278 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnj4\" (UniqueName: \"kubernetes.io/projected/8a14e049-63a2-40ed-922d-cf72de38a167-kube-api-access-spnj4\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596363 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596322 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596382 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596448 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596415 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596554 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596488 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596554 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596517 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-config\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596554 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596542 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596680 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596573 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596680 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596608 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596680 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596656 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596827 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596717 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596827 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596760 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596827 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596787 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596984 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596839 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596984 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596895 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596984 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596935 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a14e049-63a2-40ed-922d-cf72de38a167-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.596984 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.596979 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698178 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698151 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698178 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698180 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698195 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698224 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698249 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-config\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698272 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698293 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698314 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698339 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698364 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698396 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698417 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698442 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698473 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698503 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a14e049-63a2-40ed-922d-cf72de38a167-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698528 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698583 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698611 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a14e049-63a2-40ed-922d-cf72de38a167-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698634 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698641 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.698657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698659 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnj4\" (UniqueName: \"kubernetes.io/projected/8a14e049-63a2-40ed-922d-cf72de38a167-kube-api-access-spnj4\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.699461 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.698917 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.699775 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.699655 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.702234 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.702384 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.702738 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703128 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703376 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a14e049-63a2-40ed-922d-cf72de38a167-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703511 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703595 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703775 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a14e049-63a2-40ed-922d-cf72de38a167-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703934 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704028 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.703987 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-config\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704720 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.704210 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.704720 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.704576 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.705758 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.705724 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a14e049-63a2-40ed-922d-cf72de38a167-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.706330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.706305 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.706497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.706477 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.706721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.706698 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a14e049-63a2-40ed-922d-cf72de38a167-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.756776 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.756753 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnj4\" (UniqueName: \"kubernetes.io/projected/8a14e049-63a2-40ed-922d-cf72de38a167-kube-api-access-spnj4\") pod \"prometheus-k8s-0\" (UID: \"8a14e049-63a2-40ed-922d-cf72de38a167\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:55.774647 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:55.774603 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:47:56.194828 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:56.194792 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:47:56.198230 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:47:56.198203 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a14e049_63a2_40ed_922d_cf72de38a167.slice/crio-097fe55f36b8b519f20f1dbcfde54f1b299074dcbd29ef371f0601071e9c576e WatchSource:0}: Error finding container 097fe55f36b8b519f20f1dbcfde54f1b299074dcbd29ef371f0601071e9c576e: Status 404 returned error can't find the container with id 097fe55f36b8b519f20f1dbcfde54f1b299074dcbd29ef371f0601071e9c576e Oct 30 21:47:56.251524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:56.251498 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"097fe55f36b8b519f20f1dbcfde54f1b299074dcbd29ef371f0601071e9c576e"} Oct 30 21:47:56.867670 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:56.867631 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8" path="/var/lib/kubelet/pods/c64ca20a-ac6f-4eeb-8f95-9a7cc8b35cd8/volumes" Oct 30 21:47:57.257561 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:57.257530 5607 generic.go:356] "Generic (PLEG): container finished" podID="8a14e049-63a2-40ed-922d-cf72de38a167" containerID="c256023b0f855c3e51658014c9747944d84d4501938f00230e1a4c921b1761ea" exitCode=0 Oct 30 21:47:57.257982 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:57.257633 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerDied","Data":"c256023b0f855c3e51658014c9747944d84d4501938f00230e1a4c921b1761ea"} Oct 30 21:47:58.265771 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.265734 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"8d1cf85cb011148551548689d6d1b608274835b9b3fa1d607f6b3f11d0204f6b"} Oct 30 21:47:58.266309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.265838 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"6470b65b47db60d463c08ebf573249359ffc69060797435f5eadb77af7a07fa7"} Oct 30 21:47:58.266309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.265871 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"464dced3746a565e2a3b7fff2149b13a5afacae7498838ad0a7eedab6698ce3e"} Oct 30 21:47:58.266309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.265931 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"238161820ce714688cedaf0b6cbd1041740d1bc648609e825935f1cc916fe002"} Oct 30 21:47:58.266309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.265983 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"152596d9a2fcfa6f7a48c00f3ea8f3be50baa188a1272ea8d4b7f44bacf581a8"} Oct 30 21:47:58.266309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.266009 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a14e049-63a2-40ed-922d-cf72de38a167","Type":"ContainerStarted","Data":"2bfe19da68acf882b74e2292227e08f59a12d56e00c79599082ae0ad5663f37d"} Oct 30 21:47:58.395241 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:47:58.395189 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.3779891969999998 podStartE2EDuration="3.39517449s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.185293ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:58.394355093 +0000 UTC m=+57.794911216" watchObservedRunningTime="2026-10-30 21:47:58.39517449 +0000 UTC m=+57.795730604" Oct 30 21:48:00.775159 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:00.775119 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:48:06.238057 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:06.238019 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664c5b7ff9-c52hh"] Oct 30 21:48:09.867139 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:09.867103 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:48:09.885601 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:09.885574 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:48:10.331897 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:10.331874 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:48:22.863573 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:48:22.863529 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:48:31.258430 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.258374 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-664c5b7ff9-c52hh" podUID="9c952c6b-2c24-44bf-9ed2-197be86de9f8" containerName="console" containerID="cri-o://fde4530628e8ed01e9ba8914e744388a07aebf2524680a185ceede4e9d8239a6" gracePeriod=15 Oct 30 21:48:31.390938 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.390910 5607 generic.go:356] "Generic (PLEG): container finished" podID="9c952c6b-2c24-44bf-9ed2-197be86de9f8" containerID="fde4530628e8ed01e9ba8914e744388a07aebf2524680a185ceede4e9d8239a6" exitCode=0 Oct 30 21:48:31.391081 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.391008 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664c5b7ff9-c52hh" event={"ID":"9c952c6b-2c24-44bf-9ed2-197be86de9f8","Type":"ContainerDied","Data":"fde4530628e8ed01e9ba8914e744388a07aebf2524680a185ceede4e9d8239a6"} Oct 30 21:48:31.510849 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.510802 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:48:31.669072 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669043 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669211 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669092 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669211 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669137 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669211 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669186 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669383 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669222 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669383 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669283 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669383 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669330 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669383 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669370 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") pod \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\" (UID: \"9c952c6b-2c24-44bf-9ed2-197be86de9f8\") " Oct 30 21:48:31.669584 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669460 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:31.669584 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669521 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:31.669755 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669734 5607 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-config\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.669821 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669758 5607 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-oauth-serving-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.669821 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669767 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:31.669821 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669795 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:31.669821 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.669806 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:31.671725 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.671706 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:31.671896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.671882 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:31.672074 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.672056 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx" pod "9c952c6b-2c24-44bf-9ed2-197be86de9f8" (UID: "9c952c6b-2c24-44bf-9ed2-197be86de9f8"). InnerVolumeSpecName "kube-api-access-j8dvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:31.770702 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.770655 5607 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-oauth-config\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.770702 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.770676 5607 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c952c6b-2c24-44bf-9ed2-197be86de9f8-tmp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.770702 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.770686 5607 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.770702 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.770695 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8dvx\" (UniqueName: \"kubernetes.io/projected/9c952c6b-2c24-44bf-9ed2-197be86de9f8-kube-api-access-j8dvx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.770702 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.770705 5607 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c952c6b-2c24-44bf-9ed2-197be86de9f8-trusted-ca-bundle\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.770913 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:31.770714 5607 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c952c6b-2c24-44bf-9ed2-197be86de9f8-console-serving-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.397108 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:32.397076 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664c5b7ff9-c52hh" Oct 30 21:48:32.397536 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:32.397102 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664c5b7ff9-c52hh" event={"ID":"9c952c6b-2c24-44bf-9ed2-197be86de9f8","Type":"ContainerDied","Data":"52e9d887a06276b9249dff2067a0a07a6dd144c233e4ba7fd92df70292ac538b"} Oct 30 21:48:32.397536 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:32.397168 5607 scope.go:122] "RemoveContainer" containerID="fde4530628e8ed01e9ba8914e744388a07aebf2524680a185ceede4e9d8239a6" Oct 30 21:48:32.552714 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:32.552680 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664c5b7ff9-c52hh"] Oct 30 21:48:32.600382 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:32.600324 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-664c5b7ff9-c52hh"] Oct 30 21:48:32.867819 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:48:32.867777 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c952c6b-2c24-44bf-9ed2-197be86de9f8" path="/var/lib/kubelet/pods/9c952c6b-2c24-44bf-9ed2-197be86de9f8/volumes" Oct 30 21:48:52.865678 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:48:52.865636 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:49:22.865624 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:49:22.865580 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:49:52.865462 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:49:52.865415 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:50:22.861150 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:50:22.861090 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:50:52.865578 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:50:52.865533 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:51:22.861914 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:51:22.861870 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:51:44.532181 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.532087 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk"] Oct 30 21:51:44.532786 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.532496 5607 cpu_manager.go:414] "containerMap: removing container" podUID="9c952c6b-2c24-44bf-9ed2-197be86de9f8" containerName="console" Oct 30 21:51:44.532786 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.532512 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c952c6b-2c24-44bf-9ed2-197be86de9f8" containerName="console" Oct 30 21:51:44.532786 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.532568 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c952c6b-2c24-44bf-9ed2-197be86de9f8" containerName="console" Oct 30 21:51:44.535590 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.535573 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.586043 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.586019 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:51:44.598318 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.598292 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxdc\" (UniqueName: \"kubernetes.io/projected/5755b200-5a44-44b7-a94a-3d79aad54195-kube-api-access-2gxdc\") pod \"jobset-operator-674cd59dc5-zv2wk\" (UID: \"5755b200-5a44-44b7-a94a-3d79aad54195\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.598435 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.598342 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5755b200-5a44-44b7-a94a-3d79aad54195-tmp\") pod \"jobset-operator-674cd59dc5-zv2wk\" (UID: \"5755b200-5a44-44b7-a94a-3d79aad54195\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.633126 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.633104 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:51:44.633213 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.633130 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-pccfc\"" Oct 30 21:51:44.633854 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.633838 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk"] Oct 30 21:51:44.699450 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.699425 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxdc\" (UniqueName: \"kubernetes.io/projected/5755b200-5a44-44b7-a94a-3d79aad54195-kube-api-access-2gxdc\") pod \"jobset-operator-674cd59dc5-zv2wk\" (UID: \"5755b200-5a44-44b7-a94a-3d79aad54195\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.699556 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.699470 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5755b200-5a44-44b7-a94a-3d79aad54195-tmp\") pod \"jobset-operator-674cd59dc5-zv2wk\" (UID: \"5755b200-5a44-44b7-a94a-3d79aad54195\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.699772 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.699757 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5755b200-5a44-44b7-a94a-3d79aad54195-tmp\") pod \"jobset-operator-674cd59dc5-zv2wk\" (UID: \"5755b200-5a44-44b7-a94a-3d79aad54195\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.757487 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.757459 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxdc\" (UniqueName: \"kubernetes.io/projected/5755b200-5a44-44b7-a94a-3d79aad54195-kube-api-access-2gxdc\") pod \"jobset-operator-674cd59dc5-zv2wk\" (UID: \"5755b200-5a44-44b7-a94a-3d79aad54195\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:44.858563 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:44.858506 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" Oct 30 21:51:45.235539 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:45.235500 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk"] Oct 30 21:51:45.239172 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:51:45.239144 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5755b200_5a44_44b7_a94a_3d79aad54195.slice/crio-5b7d319332709d047310b70e80a37df5672b75c5f3d6d4e3fe0fa11e17425f4f WatchSource:0}: Error finding container 5b7d319332709d047310b70e80a37df5672b75c5f3d6d4e3fe0fa11e17425f4f: Status 404 returned error can't find the container with id 5b7d319332709d047310b70e80a37df5672b75c5f3d6d4e3fe0fa11e17425f4f Oct 30 21:51:46.102048 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:46.102004 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" event={"ID":"5755b200-5a44-44b7-a94a-3d79aad54195","Type":"ContainerStarted","Data":"5b7d319332709d047310b70e80a37df5672b75c5f3d6d4e3fe0fa11e17425f4f"} Oct 30 21:51:50.120796 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:50.120754 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" event={"ID":"5755b200-5a44-44b7-a94a-3d79aad54195","Type":"ContainerStarted","Data":"93a4b1682086bfe8d71a017cefa98a89c0456b86e809113adb4cb6acb9ab0995"} Oct 30 21:51:50.233428 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:51:50.233377 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-zv2wk" podStartSLOduration=2.319897984 podStartE2EDuration="6.233363698s" totalImagesPullingTime="3.913465714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:50.232517828 +0000 UTC m=+289.633073949" watchObservedRunningTime="2026-10-30 21:51:50.233363698 +0000 UTC m=+289.633919818" Oct 30 21:51:52.861722 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:51:52.861685 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:52:00.896720 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:00.896689 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:52:00.911397 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:00.911377 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:52:03.640755 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.640716 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt"] Oct 30 21:52:03.644690 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.644671 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.694532 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.694512 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:52:03.694820 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.694802 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:52:03.694896 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.694882 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sgld6\"" Oct 30 21:52:03.741383 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.741349 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:52:03.741524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.741497 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:52:03.745304 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.742014 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:52:03.750414 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.750393 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292fbd6d-ced0-4bdd-8c71-131cab596fb5-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.750501 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.750424 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/292fbd6d-ced0-4bdd-8c71-131cab596fb5-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.750549 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.750528 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kswl\" (UniqueName: \"kubernetes.io/projected/292fbd6d-ced0-4bdd-8c71-131cab596fb5-kube-api-access-7kswl\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.750586 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.750562 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.750586 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.750579 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.750675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.750654 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.759903 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.759873 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt"] Oct 30 21:52:03.851748 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.851725 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.851865 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.851752 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.851865 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.851775 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.851865 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.851802 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292fbd6d-ced0-4bdd-8c71-131cab596fb5-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.851865 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.851844 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/292fbd6d-ced0-4bdd-8c71-131cab596fb5-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.852089 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.851897 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kswl\" (UniqueName: \"kubernetes.io/projected/292fbd6d-ced0-4bdd-8c71-131cab596fb5-kube-api-access-7kswl\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.852311 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.852282 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/292fbd6d-ced0-4bdd-8c71-131cab596fb5-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.854214 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.854192 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/292fbd6d-ced0-4bdd-8c71-131cab596fb5-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.854426 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.854410 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.854553 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.854532 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.854603 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.854540 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/292fbd6d-ced0-4bdd-8c71-131cab596fb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.909593 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.909537 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kswl\" (UniqueName: \"kubernetes.io/projected/292fbd6d-ced0-4bdd-8c71-131cab596fb5-kube-api-access-7kswl\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-zjgkt\" (UID: \"292fbd6d-ced0-4bdd-8c71-131cab596fb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:03.959530 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:03.959508 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:04.448611 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:04.448577 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt"] Oct 30 21:52:04.450611 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:52:04.450580 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292fbd6d_ced0_4bdd_8c71_131cab596fb5.slice/crio-b7051c6eb852a8e7e96c0671ff3860b263203841d52fc7822b439a8283b3d6ef WatchSource:0}: Error finding container b7051c6eb852a8e7e96c0671ff3860b263203841d52fc7822b439a8283b3d6ef: Status 404 returned error can't find the container with id b7051c6eb852a8e7e96c0671ff3860b263203841d52fc7822b439a8283b3d6ef Oct 30 21:52:04.452272 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:04.452253 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:52:05.176992 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:05.176945 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" event={"ID":"292fbd6d-ced0-4bdd-8c71-131cab596fb5","Type":"ContainerStarted","Data":"b7051c6eb852a8e7e96c0671ff3860b263203841d52fc7822b439a8283b3d6ef"} Oct 30 21:52:08.190446 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:08.190413 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" event={"ID":"292fbd6d-ced0-4bdd-8c71-131cab596fb5","Type":"ContainerStarted","Data":"261d3d051f2579dea33c7881aa985bb60b8185fb73cf3a4c5229c68a253536ca"} Oct 30 21:52:09.195973 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:09.195915 5607 generic.go:356] "Generic (PLEG): container finished" podID="292fbd6d-ced0-4bdd-8c71-131cab596fb5" containerID="261d3d051f2579dea33c7881aa985bb60b8185fb73cf3a4c5229c68a253536ca" exitCode=0 Oct 30 21:52:09.196466 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:09.196006 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" event={"ID":"292fbd6d-ced0-4bdd-8c71-131cab596fb5","Type":"ContainerDied","Data":"261d3d051f2579dea33c7881aa985bb60b8185fb73cf3a4c5229c68a253536ca"} Oct 30 21:52:11.204219 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.204184 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" event={"ID":"292fbd6d-ced0-4bdd-8c71-131cab596fb5","Type":"ContainerStarted","Data":"7ceba02384b2912d2ab8c92232b7dc9709eb55992014464930effc7b74fe5a5c"} Oct 30 21:52:11.204594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.204321 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:11.490288 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.490190 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" podStartSLOduration=3.603848466 podStartE2EDuration="8.490175111s" totalImagesPullingTime="4.432897364s" totalInitContainerRuntime="453.429281ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:11.488854472 +0000 UTC m=+310.889410584" watchObservedRunningTime="2026-10-30 21:52:11.490175111 +0000 UTC m=+310.890731229" Oct 30 21:52:11.635809 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.635777 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh"] Oct 30 21:52:11.638654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.638640 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.696031 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.696008 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:52:11.696173 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.696057 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:52:11.700657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.700636 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh"] Oct 30 21:52:11.720361 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.720332 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ctg\" (UniqueName: \"kubernetes.io/projected/d74dec8a-59fa-406c-b70e-0b1a906f5777-kube-api-access-z2ctg\") pod \"cert-manager-cainjector-5c57bbd4fc-r2nwh\" (UID: \"d74dec8a-59fa-406c-b70e-0b1a906f5777\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.720494 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.720465 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d74dec8a-59fa-406c-b70e-0b1a906f5777-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r2nwh\" (UID: \"d74dec8a-59fa-406c-b70e-0b1a906f5777\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.737093 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.737077 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6dn5p\"" Oct 30 21:52:11.821331 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.821263 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ctg\" (UniqueName: \"kubernetes.io/projected/d74dec8a-59fa-406c-b70e-0b1a906f5777-kube-api-access-z2ctg\") pod \"cert-manager-cainjector-5c57bbd4fc-r2nwh\" (UID: \"d74dec8a-59fa-406c-b70e-0b1a906f5777\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.821331 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.821313 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d74dec8a-59fa-406c-b70e-0b1a906f5777-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r2nwh\" (UID: \"d74dec8a-59fa-406c-b70e-0b1a906f5777\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.898467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.898433 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d74dec8a-59fa-406c-b70e-0b1a906f5777-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-r2nwh\" (UID: \"d74dec8a-59fa-406c-b70e-0b1a906f5777\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.899572 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.899550 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ctg\" (UniqueName: \"kubernetes.io/projected/d74dec8a-59fa-406c-b70e-0b1a906f5777-kube-api-access-z2ctg\") pod \"cert-manager-cainjector-5c57bbd4fc-r2nwh\" (UID: \"d74dec8a-59fa-406c-b70e-0b1a906f5777\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.903547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.903519 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk"] Oct 30 21:52:11.906628 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.906613 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:11.949329 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.949311 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" Oct 30 21:52:11.956352 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.956333 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:52:11.956928 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.956915 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:52:11.966693 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.966662 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk"] Oct 30 21:52:11.991187 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:11.991163 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-crk8m\"" Oct 30 21:52:12.023704 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:12.023675 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zgd\" (UniqueName: \"kubernetes.io/projected/100c5340-82b8-4e93-bd2a-10ce866db411-kube-api-access-29zgd\") pod \"limitador-operator-controller-manager-5778bb4449-dfrsk\" (UID: \"100c5340-82b8-4e93-bd2a-10ce866db411\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:12.124528 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:12.124448 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zgd\" (UniqueName: \"kubernetes.io/projected/100c5340-82b8-4e93-bd2a-10ce866db411-kube-api-access-29zgd\") pod \"limitador-operator-controller-manager-5778bb4449-dfrsk\" (UID: \"100c5340-82b8-4e93-bd2a-10ce866db411\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:12.189165 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:12.189128 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zgd\" (UniqueName: \"kubernetes.io/projected/100c5340-82b8-4e93-bd2a-10ce866db411-kube-api-access-29zgd\") pod \"limitador-operator-controller-manager-5778bb4449-dfrsk\" (UID: \"100c5340-82b8-4e93-bd2a-10ce866db411\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:12.220648 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:12.220621 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:12.317855 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:12.317828 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh"] Oct 30 21:52:12.320246 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:52:12.320221 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd74dec8a_59fa_406c_b70e_0b1a906f5777.slice/crio-71ad30d9168a16c0ab532b188a66b657ffcc9c52155dbf1207750084ea3c32a4 WatchSource:0}: Error finding container 71ad30d9168a16c0ab532b188a66b657ffcc9c52155dbf1207750084ea3c32a4: Status 404 returned error can't find the container with id 71ad30d9168a16c0ab532b188a66b657ffcc9c52155dbf1207750084ea3c32a4 Oct 30 21:52:12.588793 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:12.588773 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk"] Oct 30 21:52:12.591396 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:52:12.591364 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100c5340_82b8_4e93_bd2a_10ce866db411.slice/crio-bdc25f466272a5de9e009e687669a88b2c2246485c64c04f726315d6fb02137d WatchSource:0}: Error finding container bdc25f466272a5de9e009e687669a88b2c2246485c64c04f726315d6fb02137d: Status 404 returned error can't find the container with id bdc25f466272a5de9e009e687669a88b2c2246485c64c04f726315d6fb02137d Oct 30 21:52:13.214139 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.214101 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" event={"ID":"100c5340-82b8-4e93-bd2a-10ce866db411","Type":"ContainerStarted","Data":"bdc25f466272a5de9e009e687669a88b2c2246485c64c04f726315d6fb02137d"} Oct 30 21:52:13.215313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.215287 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" event={"ID":"d74dec8a-59fa-406c-b70e-0b1a906f5777","Type":"ContainerStarted","Data":"71ad30d9168a16c0ab532b188a66b657ffcc9c52155dbf1207750084ea3c32a4"} Oct 30 21:52:13.595697 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.595628 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5"] Oct 30 21:52:13.602026 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.602006 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.653863 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.653841 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5"] Oct 30 21:52:13.699545 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.699520 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-p7nxz\"" Oct 30 21:52:13.738728 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.738703 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0028b4a-c18c-4239-9c6c-d4f604efb739-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5jzd5\" (UID: \"d0028b4a-c18c-4239-9c6c-d4f604efb739\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.738835 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.738746 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8g55\" (UniqueName: \"kubernetes.io/projected/d0028b4a-c18c-4239-9c6c-d4f604efb739-kube-api-access-h8g55\") pod \"cert-manager-webhook-5dbbbf75cc-5jzd5\" (UID: \"d0028b4a-c18c-4239-9c6c-d4f604efb739\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.839836 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.839808 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8g55\" (UniqueName: \"kubernetes.io/projected/d0028b4a-c18c-4239-9c6c-d4f604efb739-kube-api-access-h8g55\") pod \"cert-manager-webhook-5dbbbf75cc-5jzd5\" (UID: \"d0028b4a-c18c-4239-9c6c-d4f604efb739\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.839977 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.839874 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0028b4a-c18c-4239-9c6c-d4f604efb739-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5jzd5\" (UID: \"d0028b4a-c18c-4239-9c6c-d4f604efb739\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.902325 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.902273 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0028b4a-c18c-4239-9c6c-d4f604efb739-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5jzd5\" (UID: \"d0028b4a-c18c-4239-9c6c-d4f604efb739\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.902652 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.902634 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8g55\" (UniqueName: \"kubernetes.io/projected/d0028b4a-c18c-4239-9c6c-d4f604efb739-kube-api-access-h8g55\") pod \"cert-manager-webhook-5dbbbf75cc-5jzd5\" (UID: \"d0028b4a-c18c-4239-9c6c-d4f604efb739\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:13.913458 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:13.913440 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:14.275589 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:14.275559 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5"] Oct 30 21:52:14.279120 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:52:14.279084 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0028b4a_c18c_4239_9c6c_d4f604efb739.slice/crio-f3fea7ec4bc314b0fbb25b50531acd7a4c41b1fbbfda249b9baa4eb878278c19 WatchSource:0}: Error finding container f3fea7ec4bc314b0fbb25b50531acd7a4c41b1fbbfda249b9baa4eb878278c19: Status 404 returned error can't find the container with id f3fea7ec4bc314b0fbb25b50531acd7a4c41b1fbbfda249b9baa4eb878278c19 Oct 30 21:52:15.228451 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:15.228392 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" event={"ID":"d0028b4a-c18c-4239-9c6c-d4f604efb739","Type":"ContainerStarted","Data":"f3fea7ec4bc314b0fbb25b50531acd7a4c41b1fbbfda249b9baa4eb878278c19"} Oct 30 21:52:17.242092 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:17.242042 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" event={"ID":"100c5340-82b8-4e93-bd2a-10ce866db411","Type":"ContainerStarted","Data":"ae2ecfd84a7b1fde433d8347adbc97f82db9a08d8225f009bd1e45744181b1c2"} Oct 30 21:52:17.242487 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:17.242193 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:17.355023 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:17.354881 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" podStartSLOduration=1.8794089760000001 podStartE2EDuration="6.354859808s" totalImagesPullingTime="4.475450832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:17.353772869 +0000 UTC m=+316.754328990" watchObservedRunningTime="2026-10-30 21:52:17.354859808 +0000 UTC m=+316.755415930" Oct 30 21:52:18.247018 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:18.246975 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" event={"ID":"d74dec8a-59fa-406c-b70e-0b1a906f5777","Type":"ContainerStarted","Data":"9c8072a1099d4df977de516b1aa3cd726d94bacd1a3be23e11c73a9b217db5e9"} Oct 30 21:52:18.356547 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:18.356500 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-r2nwh" podStartSLOduration=2.016638941 podStartE2EDuration="7.356486149s" totalImagesPullingTime="5.339847208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:18.355776582 +0000 UTC m=+317.756332697" watchObservedRunningTime="2026-10-30 21:52:18.356486149 +0000 UTC m=+317.757042271" Oct 30 21:52:19.253122 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:19.253084 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" event={"ID":"d0028b4a-c18c-4239-9c6c-d4f604efb739","Type":"ContainerStarted","Data":"2aa50b0e7f076d51bd7feb83a4c31528920aa4516c4f3ea06f20f1c482b3acf8"} Oct 30 21:52:19.253525 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:19.253222 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:19.362932 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:19.362863 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" podStartSLOduration=2.363038242 podStartE2EDuration="6.362848877s" totalImagesPullingTime="3.999810635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:19.361905595 +0000 UTC m=+318.762461716" watchObservedRunningTime="2026-10-30 21:52:19.362848877 +0000 UTC m=+318.763404997" Oct 30 21:52:22.210722 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:22.210690 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-zjgkt" Oct 30 21:52:22.862823 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:52:22.862791 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:52:25.259274 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:25.259235 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5jzd5" Oct 30 21:52:28.249207 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:28.249175 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dfrsk" Oct 30 21:52:36.168931 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.168897 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d969c5987-s82hb"] Oct 30 21:52:36.178491 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.178467 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.219144 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.219112 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw744\" (UniqueName: \"kubernetes.io/projected/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-kube-api-access-fw744\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.219271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.219219 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-tmp\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.219271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.219258 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-tls-certs\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.227836 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.227817 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d969c5987-s82hb"] Oct 30 21:52:36.238892 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.238872 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:52:36.238892 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.238872 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:52:36.273619 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.273599 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:52:36.273744 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.273672 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p6chm\"" Oct 30 21:52:36.273744 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.273689 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:52:36.320669 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.320641 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-tmp\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.320829 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.320698 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-tls-certs\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.320893 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.320830 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw744\" (UniqueName: \"kubernetes.io/projected/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-kube-api-access-fw744\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.325998 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.324019 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-tmp\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.325998 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.324803 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-tls-certs\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.379141 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.379118 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw744\" (UniqueName: \"kubernetes.io/projected/d893d4a3-eb6d-47c3-a12c-10e1e99be3d6-kube-api-access-fw744\") pod \"kube-auth-proxy-6d969c5987-s82hb\" (UID: \"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.489666 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.489647 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" Oct 30 21:52:36.853215 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:36.853190 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d969c5987-s82hb"] Oct 30 21:52:36.855342 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:52:36.855313 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd893d4a3_eb6d_47c3_a12c_10e1e99be3d6.slice/crio-2b62fab141767d1ab4dfccd61385d824cbd98be717a74668a5c0a985adc9bb66 WatchSource:0}: Error finding container 2b62fab141767d1ab4dfccd61385d824cbd98be717a74668a5c0a985adc9bb66: Status 404 returned error can't find the container with id 2b62fab141767d1ab4dfccd61385d824cbd98be717a74668a5c0a985adc9bb66 Oct 30 21:52:37.328758 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:37.328713 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" event={"ID":"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6","Type":"ContainerStarted","Data":"2b62fab141767d1ab4dfccd61385d824cbd98be717a74668a5c0a985adc9bb66"} Oct 30 21:52:40.343678 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:40.343644 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" event={"ID":"d893d4a3-eb6d-47c3-a12c-10e1e99be3d6","Type":"ContainerStarted","Data":"dbba7572b4e28f56c075c9b63d48b722299c0a879d6bbccdebe24bac0ffcafa2"} Oct 30 21:52:40.455591 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:40.455546 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d969c5987-s82hb" podStartSLOduration=1.5690994809999999 podStartE2EDuration="4.455531297s" totalImagesPullingTime="2.886431816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:40.454285129 +0000 UTC m=+339.854841251" watchObservedRunningTime="2026-10-30 21:52:40.455531297 +0000 UTC m=+339.856087418" Oct 30 21:52:43.670317 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.670276 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h7zm2"] Oct 30 21:52:43.673654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.673635 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:52:43.725055 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.725014 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h7zm2"] Oct 30 21:52:43.770681 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.770660 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b7wsj\"" Oct 30 21:52:43.775556 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.775534 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4nh\" (UniqueName: \"kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh\") pod \"authorino-56c6c9ddd5-h7zm2\" (UID: \"e73f627d-6a82-4018-8875-aa540156edcf\") " pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:52:43.876309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.876288 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4nh\" (UniqueName: \"kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh\") pod \"authorino-56c6c9ddd5-h7zm2\" (UID: \"e73f627d-6a82-4018-8875-aa540156edcf\") " pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:52:43.934664 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.934590 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4nh\" (UniqueName: \"kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh\") pod \"authorino-56c6c9ddd5-h7zm2\" (UID: \"e73f627d-6a82-4018-8875-aa540156edcf\") " pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:52:43.985241 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:43.985219 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:52:44.344462 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:44.344435 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h7zm2"] Oct 30 21:52:44.346833 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:52:44.346798 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73f627d_6a82_4018_8875_aa540156edcf.slice/crio-fb3baf8ba51f13e661b848aadad2a46ef6618afaebe295a440b1f93ceb75d247 WatchSource:0}: Error finding container fb3baf8ba51f13e661b848aadad2a46ef6618afaebe295a440b1f93ceb75d247: Status 404 returned error can't find the container with id fb3baf8ba51f13e661b848aadad2a46ef6618afaebe295a440b1f93ceb75d247 Oct 30 21:52:44.361216 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:44.361188 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" event={"ID":"e73f627d-6a82-4018-8875-aa540156edcf","Type":"ContainerStarted","Data":"fb3baf8ba51f13e661b848aadad2a46ef6618afaebe295a440b1f93ceb75d247"} Oct 30 21:52:47.375904 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:47.375837 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" event={"ID":"e73f627d-6a82-4018-8875-aa540156edcf","Type":"ContainerStarted","Data":"308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9"} Oct 30 21:52:47.484201 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:52:47.484156 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" podStartSLOduration=1.720255951 podStartE2EDuration="4.484143421s" totalImagesPullingTime="2.76388747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:47.483405243 +0000 UTC m=+346.883961364" watchObservedRunningTime="2026-10-30 21:52:47.484143421 +0000 UTC m=+346.884699543" Oct 30 21:52:52.861813 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:52:52.861766 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:53:01.014988 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.014944 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479"] Oct 30 21:53:01.021599 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.021580 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.071357 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.071334 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:53:01.107462 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.107437 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.107588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.107467 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.107588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.107485 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9pxt\" (UniqueName: \"kubernetes.io/projected/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-kube-api-access-x9pxt\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.107588 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.107504 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.117867 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.117843 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pgl7b\"" Oct 30 21:53:01.118008 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.117866 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:53:01.208045 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.208023 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.208169 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.208057 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.208169 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.208074 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9pxt\" (UniqueName: \"kubernetes.io/projected/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-kube-api-access-x9pxt\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.208169 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.208103 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.208388 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.208369 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.208441 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.208421 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.212218 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.212187 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 21:53:01.213313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.213298 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479"] Oct 30 21:53:01.219194 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.219177 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.266463 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.266409 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9pxt\" (UniqueName: \"kubernetes.io/projected/525b6cdb-6e46-470f-b8a6-9fde60d3cb3f-kube-api-access-x9pxt\") pod \"kserve-module-controller-manager-7566cf5fc8-rp479\" (UID: \"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.333529 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.333505 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:01.791013 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:01.790979 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479"] Oct 30 21:53:01.794479 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:53:01.794446 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525b6cdb_6e46_470f_b8a6_9fde60d3cb3f.slice/crio-e815e17e163e7f2709b81ca294ed9ccc4a332de37b8b0e2b357a2f5e061ff013 WatchSource:0}: Error finding container e815e17e163e7f2709b81ca294ed9ccc4a332de37b8b0e2b357a2f5e061ff013: Status 404 returned error can't find the container with id e815e17e163e7f2709b81ca294ed9ccc4a332de37b8b0e2b357a2f5e061ff013 Oct 30 21:53:02.440832 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:02.440792 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" event={"ID":"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f","Type":"ContainerStarted","Data":"e815e17e163e7f2709b81ca294ed9ccc4a332de37b8b0e2b357a2f5e061ff013"} Oct 30 21:53:04.451745 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:04.451709 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" event={"ID":"525b6cdb-6e46-470f-b8a6-9fde60d3cb3f","Type":"ContainerStarted","Data":"2ed04868566e7fee864f2f75bda736b011f6c0966d8055ef57b8078ba3bcec5a"} Oct 30 21:53:04.452175 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:04.451843 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:04.617039 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:04.616994 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" podStartSLOduration=2.65049595 podStartE2EDuration="4.616979849s" totalImagesPullingTime="1.966483899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:04.616105058 +0000 UTC m=+364.016661180" watchObservedRunningTime="2026-10-30 21:53:04.616979849 +0000 UTC m=+364.017535991" Oct 30 21:53:07.712920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.712892 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q"] Oct 30 21:53:07.716290 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.716275 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:07.763440 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.763417 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc26d\" (UniqueName: \"kubernetes.io/projected/59bfd83b-c3e4-4e10-b631-b12d8ffbf728-kube-api-access-nc26d\") pod \"postgres-7cdbf6dc44-gvp8q\" (UID: \"59bfd83b-c3e4-4e10-b631-b12d8ffbf728\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:07.763564 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.763541 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59bfd83b-c3e4-4e10-b631-b12d8ffbf728-data\") pod \"postgres-7cdbf6dc44-gvp8q\" (UID: \"59bfd83b-c3e4-4e10-b631-b12d8ffbf728\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:07.768371 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.768349 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:07.789031 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.789013 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:07.812788 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.812768 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 21:53:07.812933 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.812916 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sttkj\"" Oct 30 21:53:07.813927 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.813911 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q"] Oct 30 21:53:07.864055 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.864030 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59bfd83b-c3e4-4e10-b631-b12d8ffbf728-data\") pod \"postgres-7cdbf6dc44-gvp8q\" (UID: \"59bfd83b-c3e4-4e10-b631-b12d8ffbf728\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:07.864154 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.864088 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc26d\" (UniqueName: \"kubernetes.io/projected/59bfd83b-c3e4-4e10-b631-b12d8ffbf728-kube-api-access-nc26d\") pod \"postgres-7cdbf6dc44-gvp8q\" (UID: \"59bfd83b-c3e4-4e10-b631-b12d8ffbf728\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:07.864376 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.864359 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/59bfd83b-c3e4-4e10-b631-b12d8ffbf728-data\") pod \"postgres-7cdbf6dc44-gvp8q\" (UID: \"59bfd83b-c3e4-4e10-b631-b12d8ffbf728\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:07.923014 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:07.922989 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc26d\" (UniqueName: \"kubernetes.io/projected/59bfd83b-c3e4-4e10-b631-b12d8ffbf728-kube-api-access-nc26d\") pod \"postgres-7cdbf6dc44-gvp8q\" (UID: \"59bfd83b-c3e4-4e10-b631-b12d8ffbf728\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:08.028464 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:08.028402 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:08.392577 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:08.392554 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q"] Oct 30 21:53:08.395052 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:53:08.395026 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bfd83b_c3e4_4e10_b631_b12d8ffbf728.slice/crio-6e4b5118d39a464983cd87fd9240211fe771ca4ce62a70ee5f488dbb8f02b860 WatchSource:0}: Error finding container 6e4b5118d39a464983cd87fd9240211fe771ca4ce62a70ee5f488dbb8f02b860: Status 404 returned error can't find the container with id 6e4b5118d39a464983cd87fd9240211fe771ca4ce62a70ee5f488dbb8f02b860 Oct 30 21:53:08.469752 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:08.469725 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" event={"ID":"59bfd83b-c3e4-4e10-b631-b12d8ffbf728","Type":"ContainerStarted","Data":"6e4b5118d39a464983cd87fd9240211fe771ca4ce62a70ee5f488dbb8f02b860"} Oct 30 21:53:15.508072 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:15.508039 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" event={"ID":"59bfd83b-c3e4-4e10-b631-b12d8ffbf728","Type":"ContainerStarted","Data":"97d3dc242a44adf5d5eb12191adf5410e7a03c3047dc08478fa8776db15bc62b"} Oct 30 21:53:15.508524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:15.508104 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:15.618856 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:15.618664 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" podStartSLOduration=2.086925166 podStartE2EDuration="8.618646549s" totalImagesPullingTime="6.531721383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:15.618463881 +0000 UTC m=+375.019020007" watchObservedRunningTime="2026-10-30 21:53:15.618646549 +0000 UTC m=+375.019202671" Oct 30 21:53:21.544094 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:21.544065 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gvp8q" Oct 30 21:53:22.863892 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:53:22.863851 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:53:25.457922 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:53:25.457892 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-rp479" Oct 30 21:53:52.864191 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:53:52.864151 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:54:07.377611 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.377578 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pwlj6"] Oct 30 21:54:07.402530 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.402506 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.436708 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.436683 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pwlj6"] Oct 30 21:54:07.443654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.443634 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7308c1fa-b891-4a24-b173-cd6a9d361851-cert\") pod \"kserve-controller-manager-776cc66d47-pwlj6\" (UID: \"7308c1fa-b891-4a24-b173-cd6a9d361851\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.443763 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.443669 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbg5\" (UniqueName: \"kubernetes.io/projected/7308c1fa-b891-4a24-b173-cd6a9d361851-kube-api-access-7rbg5\") pod \"kserve-controller-manager-776cc66d47-pwlj6\" (UID: \"7308c1fa-b891-4a24-b173-cd6a9d361851\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.482380 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.482362 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:54:07.508416 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.508391 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wdgjb\"" Oct 30 21:54:07.544484 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.544464 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7308c1fa-b891-4a24-b173-cd6a9d361851-cert\") pod \"kserve-controller-manager-776cc66d47-pwlj6\" (UID: \"7308c1fa-b891-4a24-b173-cd6a9d361851\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.544575 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.544495 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbg5\" (UniqueName: \"kubernetes.io/projected/7308c1fa-b891-4a24-b173-cd6a9d361851-kube-api-access-7rbg5\") pod \"kserve-controller-manager-776cc66d47-pwlj6\" (UID: \"7308c1fa-b891-4a24-b173-cd6a9d361851\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.546911 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.546889 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7308c1fa-b891-4a24-b173-cd6a9d361851-cert\") pod \"kserve-controller-manager-776cc66d47-pwlj6\" (UID: \"7308c1fa-b891-4a24-b173-cd6a9d361851\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.609652 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.609625 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbg5\" (UniqueName: \"kubernetes.io/projected/7308c1fa-b891-4a24-b173-cd6a9d361851-kube-api-access-7rbg5\") pod \"kserve-controller-manager-776cc66d47-pwlj6\" (UID: \"7308c1fa-b891-4a24-b173-cd6a9d361851\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:07.714911 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:07.714891 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:08.079824 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:08.079798 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pwlj6"] Oct 30 21:54:08.082553 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:54:08.082520 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7308c1fa_b891_4a24_b173_cd6a9d361851.slice/crio-da4e4de0f4971c7f96e71d603555bf6c7f4fd31dab8003e6c17eff12bb141073 WatchSource:0}: Error finding container da4e4de0f4971c7f96e71d603555bf6c7f4fd31dab8003e6c17eff12bb141073: Status 404 returned error can't find the container with id da4e4de0f4971c7f96e71d603555bf6c7f4fd31dab8003e6c17eff12bb141073 Oct 30 21:54:08.748790 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:08.748752 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" event={"ID":"7308c1fa-b891-4a24-b173-cd6a9d361851","Type":"ContainerStarted","Data":"da4e4de0f4971c7f96e71d603555bf6c7f4fd31dab8003e6c17eff12bb141073"} Oct 30 21:54:10.759266 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:10.759238 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" event={"ID":"7308c1fa-b891-4a24-b173-cd6a9d361851","Type":"ContainerStarted","Data":"a7e78a3c06925758efdcbcb317522ae5d3a26ed78e04a9a392dac2e4a0b2ec44"} Oct 30 21:54:10.759634 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:10.759341 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:10.874715 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:10.874627 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" podStartSLOduration=1.333987599 podStartE2EDuration="3.87460977s" totalImagesPullingTime="2.540622171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.872853183 +0000 UTC m=+430.273409303" watchObservedRunningTime="2026-10-30 21:54:10.87460977 +0000 UTC m=+430.275165888" Oct 30 21:54:22.862785 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:54:22.862740 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:54:24.766071 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:24.766039 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69c469695-qxqk8"] Oct 30 21:54:24.769396 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:24.769379 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:24.825252 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:24.825227 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69c469695-qxqk8"] Oct 30 21:54:24.885579 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:24.885558 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qcr\" (UniqueName: \"kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr\") pod \"authorino-69c469695-qxqk8\" (UID: \"47805953-778a-4518-bd01-f3c20c99870b\") " pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:24.986162 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:24.986136 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qcr\" (UniqueName: \"kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr\") pod \"authorino-69c469695-qxqk8\" (UID: \"47805953-778a-4518-bd01-f3c20c99870b\") " pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:25.053699 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:25.053636 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qcr\" (UniqueName: \"kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr\") pod \"authorino-69c469695-qxqk8\" (UID: \"47805953-778a-4518-bd01-f3c20c99870b\") " pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:25.080313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:25.080292 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:25.164008 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:25.163978 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69c469695-qxqk8"] Oct 30 21:54:25.442797 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:25.442685 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69c469695-qxqk8"] Oct 30 21:54:25.445499 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:54:25.445471 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47805953_778a_4518_bd01_f3c20c99870b.slice/crio-57591a52d374b80111b60be28470e0661d91a758cf2822225208607d88948ced WatchSource:0}: Error finding container 57591a52d374b80111b60be28470e0661d91a758cf2822225208607d88948ced: Status 404 returned error can't find the container with id 57591a52d374b80111b60be28470e0661d91a758cf2822225208607d88948ced Oct 30 21:54:25.827926 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:25.827847 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69c469695-qxqk8" event={"ID":"47805953-778a-4518-bd01-f3c20c99870b","Type":"ContainerStarted","Data":"57591a52d374b80111b60be28470e0661d91a758cf2822225208607d88948ced"} Oct 30 21:54:27.838294 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:27.838253 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69c469695-qxqk8" event={"ID":"47805953-778a-4518-bd01-f3c20c99870b","Type":"ContainerStarted","Data":"4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0"} Oct 30 21:54:27.838738 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:27.838331 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-69c469695-qxqk8" podUID="47805953-778a-4518-bd01-f3c20c99870b" containerName="authorino" containerID="cri-o://4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0" gracePeriod=30 Oct 30 21:54:27.957606 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:27.957557 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69c469695-qxqk8" podStartSLOduration=2.510562261 podStartE2EDuration="3.957542418s" totalImagesPullingTime="1.446980157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:27.956774641 +0000 UTC m=+447.357330759" watchObservedRunningTime="2026-10-30 21:54:27.957542418 +0000 UTC m=+447.358098536" Oct 30 21:54:28.107968 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.107925 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:28.210505 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.210473 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr\" (UniqueName: \"kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr\") pod \"47805953-778a-4518-bd01-f3c20c99870b\" (UID: \"47805953-778a-4518-bd01-f3c20c99870b\") " Oct 30 21:54:28.212822 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.212797 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr" pod "47805953-778a-4518-bd01-f3c20c99870b" (UID: "47805953-778a-4518-bd01-f3c20c99870b"). InnerVolumeSpecName "kube-api-access-45qcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:28.311629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.311597 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qcr\" (UniqueName: \"kubernetes.io/projected/47805953-778a-4518-bd01-f3c20c99870b-kube-api-access-45qcr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.736361 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.736328 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h7zm2"] Oct 30 21:54:28.736537 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.736517 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" podUID="e73f627d-6a82-4018-8875-aa540156edcf" containerName="authorino" containerID="cri-o://308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9" gracePeriod=30 Oct 30 21:54:28.844133 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.844102 5607 generic.go:356] "Generic (PLEG): container finished" podID="47805953-778a-4518-bd01-f3c20c99870b" containerID="4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0" exitCode=0 Oct 30 21:54:28.844479 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.844149 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69c469695-qxqk8" Oct 30 21:54:28.844479 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.844182 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69c469695-qxqk8" event={"ID":"47805953-778a-4518-bd01-f3c20c99870b","Type":"ContainerDied","Data":"4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0"} Oct 30 21:54:28.844479 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.844227 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69c469695-qxqk8" event={"ID":"47805953-778a-4518-bd01-f3c20c99870b","Type":"ContainerDied","Data":"57591a52d374b80111b60be28470e0661d91a758cf2822225208607d88948ced"} Oct 30 21:54:28.844479 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.844246 5607 scope.go:122] "RemoveContainer" containerID="4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0" Oct 30 21:54:28.853542 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.853443 5607 scope.go:122] "RemoveContainer" containerID="4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0" Oct 30 21:54:28.853777 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:54:28.853756 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0\": container with ID starting with 4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0 not found: ID does not exist" containerID="4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0" Oct 30 21:54:28.853857 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.853789 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0"} err="failed to get container status \"4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0\": rpc error: code = NotFound desc = could not find container \"4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0\": container with ID starting with 4505f77adf4ad22c4acea687715ce1a26bf748dc1ebec09d624a8779a5412fe0 not found: ID does not exist" Oct 30 21:54:28.973509 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:28.973490 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:54:29.016117 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.016054 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69c469695-qxqk8"] Oct 30 21:54:29.018470 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.018448 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69c469695-qxqk8"] Oct 30 21:54:29.121040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.121020 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh\" (UniqueName: \"kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh\") pod \"e73f627d-6a82-4018-8875-aa540156edcf\" (UID: \"e73f627d-6a82-4018-8875-aa540156edcf\") " Oct 30 21:54:29.123136 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.123110 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh" pod "e73f627d-6a82-4018-8875-aa540156edcf" (UID: "e73f627d-6a82-4018-8875-aa540156edcf"). InnerVolumeSpecName "kube-api-access-pj4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:29.221816 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.221794 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4nh\" (UniqueName: \"kubernetes.io/projected/e73f627d-6a82-4018-8875-aa540156edcf-kube-api-access-pj4nh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:29.851362 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.851331 5607 generic.go:356] "Generic (PLEG): container finished" podID="e73f627d-6a82-4018-8875-aa540156edcf" containerID="308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9" exitCode=0 Oct 30 21:54:29.851830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.851385 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" Oct 30 21:54:29.851830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.851423 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" event={"ID":"e73f627d-6a82-4018-8875-aa540156edcf","Type":"ContainerDied","Data":"308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9"} Oct 30 21:54:29.851830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.851473 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h7zm2" event={"ID":"e73f627d-6a82-4018-8875-aa540156edcf","Type":"ContainerDied","Data":"fb3baf8ba51f13e661b848aadad2a46ef6618afaebe295a440b1f93ceb75d247"} Oct 30 21:54:29.851830 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.851505 5607 scope.go:122] "RemoveContainer" containerID="308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9" Oct 30 21:54:29.861370 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.861349 5607 scope.go:122] "RemoveContainer" containerID="308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9" Oct 30 21:54:29.861680 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:54:29.861660 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9\": container with ID starting with 308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9 not found: ID does not exist" containerID="308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9" Oct 30 21:54:29.861724 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:29.861691 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9"} err="failed to get container status \"308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9\": rpc error: code = NotFound desc = could not find container \"308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9\": container with ID starting with 308911eb4010778d5891d7d87e8d1d24745cd562f997524186e170e3212eb3a9 not found: ID does not exist" Oct 30 21:54:30.006487 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:30.006461 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h7zm2"] Oct 30 21:54:30.008939 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:30.008917 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h7zm2"] Oct 30 21:54:30.869037 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:30.869009 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47805953-778a-4518-bd01-f3c20c99870b" path="/var/lib/kubelet/pods/47805953-778a-4518-bd01-f3c20c99870b/volumes" Oct 30 21:54:30.869429 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:30.869312 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73f627d-6a82-4018-8875-aa540156edcf" path="/var/lib/kubelet/pods/e73f627d-6a82-4018-8875-aa540156edcf/volumes" Oct 30 21:54:41.769342 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:54:41.769235 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pwlj6" Oct 30 21:54:52.864920 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:54:52.864875 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:55:22.865813 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:55:22.865768 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:55:52.866288 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:55:52.866173 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:56:04.154227 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154195 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-bjh6x"] Oct 30 21:56:04.154675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154525 5607 cpu_manager.go:414] "containerMap: removing container" podUID="47805953-778a-4518-bd01-f3c20c99870b" containerName="authorino" Oct 30 21:56:04.154675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154535 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47805953-778a-4518-bd01-f3c20c99870b" containerName="authorino" Oct 30 21:56:04.154675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154551 5607 cpu_manager.go:414] "containerMap: removing container" podUID="e73f627d-6a82-4018-8875-aa540156edcf" containerName="authorino" Oct 30 21:56:04.154675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154556 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73f627d-6a82-4018-8875-aa540156edcf" containerName="authorino" Oct 30 21:56:04.154675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154608 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="47805953-778a-4518-bd01-f3c20c99870b" containerName="authorino" Oct 30 21:56:04.154675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.154618 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73f627d-6a82-4018-8875-aa540156edcf" containerName="authorino" Oct 30 21:56:04.157536 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.157520 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.207258 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.207239 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:56:04.207555 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.207541 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:56:04.253906 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.253884 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-vxfgf\"" Oct 30 21:56:04.254891 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.254872 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-bjh6x"] Oct 30 21:56:04.341692 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.341671 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-bjh6x\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.341797 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.341736 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q6qt\" (UniqueName: \"kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt\") pod \"ai-gateway-controller-7d85675f98-bjh6x\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.442331 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.442309 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-bjh6x\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.442424 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.442364 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q6qt\" (UniqueName: \"kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt\") pod \"ai-gateway-controller-7d85675f98-bjh6x\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.445112 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.445089 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-bjh6x\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.500827 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.500806 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q6qt\" (UniqueName: \"kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt\") pod \"ai-gateway-controller-7d85675f98-bjh6x\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:04.769620 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:04.769562 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:05.133544 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:05.133517 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-bjh6x"] Oct 30 21:56:05.135633 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:56:05.135604 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb95dbc_b350_4263_aa46_26d464d029a5.slice/crio-88b6a263dfd5e1bdaab09452057ca0372e042f3e76c567b153bea19092460250 WatchSource:0}: Error finding container 88b6a263dfd5e1bdaab09452057ca0372e042f3e76c567b153bea19092460250: Status 404 returned error can't find the container with id 88b6a263dfd5e1bdaab09452057ca0372e042f3e76c567b153bea19092460250 Oct 30 21:56:05.269674 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:05.269643 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" event={"ID":"3bb95dbc-b350-4263-aa46-26d464d029a5","Type":"ContainerStarted","Data":"88b6a263dfd5e1bdaab09452057ca0372e042f3e76c567b153bea19092460250"} Oct 30 21:56:09.293035 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:09.293002 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" event={"ID":"3bb95dbc-b350-4263-aa46-26d464d029a5","Type":"ContainerStarted","Data":"45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f"} Oct 30 21:56:09.293498 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:09.293068 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:09.403974 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:09.403904 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" podStartSLOduration=2.103968989 podStartE2EDuration="5.403888587s" totalImagesPullingTime="3.299919598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:09.40343543 +0000 UTC m=+548.803991551" watchObservedRunningTime="2026-10-30 21:56:09.403888587 +0000 UTC m=+548.804444708" Oct 30 21:56:12.826066 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.825987 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2"] Oct 30 21:56:12.829315 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.829300 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:12.885785 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.885765 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-957v7\"" Oct 30 21:56:12.932805 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.932784 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:56:12.933826 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.933807 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2"] Oct 30 21:56:12.980648 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.980624 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8cgnk"] Oct 30 21:56:12.983902 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:12.983886 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.012809 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.012789 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llr7\" (UniqueName: \"kubernetes.io/projected/1ff003b9-d2ce-4ba0-b05a-5cca5c076893-kube-api-access-2llr7\") pod \"payload-pre-processing-7b98bb678c-9xpg2\" (UID: \"1ff003b9-d2ce-4ba0-b05a-5cca5c076893\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.012910 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.012829 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff003b9-d2ce-4ba0-b05a-5cca5c076893-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9xpg2\" (UID: \"1ff003b9-d2ce-4ba0-b05a-5cca5c076893\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.035871 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.035847 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8cgnk"] Oct 30 21:56:13.113464 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.113411 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llr7\" (UniqueName: \"kubernetes.io/projected/1ff003b9-d2ce-4ba0-b05a-5cca5c076893-kube-api-access-2llr7\") pod \"payload-pre-processing-7b98bb678c-9xpg2\" (UID: \"1ff003b9-d2ce-4ba0-b05a-5cca5c076893\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.113464 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.113451 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff003b9-d2ce-4ba0-b05a-5cca5c076893-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9xpg2\" (UID: \"1ff003b9-d2ce-4ba0-b05a-5cca5c076893\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.113613 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.113517 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2972d9e0-bf8e-4dc6-b7b8-25caee856f57-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-8cgnk\" (UID: \"2972d9e0-bf8e-4dc6-b7b8-25caee856f57\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.113613 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.113554 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlv2w\" (UniqueName: \"kubernetes.io/projected/2972d9e0-bf8e-4dc6-b7b8-25caee856f57-kube-api-access-dlv2w\") pod \"payload-processing-55fbbbf9dc-8cgnk\" (UID: \"2972d9e0-bf8e-4dc6-b7b8-25caee856f57\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.114277 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.114260 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff003b9-d2ce-4ba0-b05a-5cca5c076893-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9xpg2\" (UID: \"1ff003b9-d2ce-4ba0-b05a-5cca5c076893\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.172309 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.172288 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llr7\" (UniqueName: \"kubernetes.io/projected/1ff003b9-d2ce-4ba0-b05a-5cca5c076893-kube-api-access-2llr7\") pod \"payload-pre-processing-7b98bb678c-9xpg2\" (UID: \"1ff003b9-d2ce-4ba0-b05a-5cca5c076893\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.214122 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.214096 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2972d9e0-bf8e-4dc6-b7b8-25caee856f57-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-8cgnk\" (UID: \"2972d9e0-bf8e-4dc6-b7b8-25caee856f57\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.214212 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.214143 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlv2w\" (UniqueName: \"kubernetes.io/projected/2972d9e0-bf8e-4dc6-b7b8-25caee856f57-kube-api-access-dlv2w\") pod \"payload-processing-55fbbbf9dc-8cgnk\" (UID: \"2972d9e0-bf8e-4dc6-b7b8-25caee856f57\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.214759 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.214737 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2972d9e0-bf8e-4dc6-b7b8-25caee856f57-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-8cgnk\" (UID: \"2972d9e0-bf8e-4dc6-b7b8-25caee856f57\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.272839 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.272820 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlv2w\" (UniqueName: \"kubernetes.io/projected/2972d9e0-bf8e-4dc6-b7b8-25caee856f57-kube-api-access-dlv2w\") pod \"payload-processing-55fbbbf9dc-8cgnk\" (UID: \"2972d9e0-bf8e-4dc6-b7b8-25caee856f57\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.295634 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.295610 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:13.441001 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.440975 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:13.665797 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.665769 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8cgnk"] Oct 30 21:56:13.667764 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:56:13.667735 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2972d9e0_bf8e_4dc6_b7b8_25caee856f57.slice/crio-0263b6de560cb4e0afe12fab013438b421a3e93c7e9956c55cc92fbe687715e7 WatchSource:0}: Error finding container 0263b6de560cb4e0afe12fab013438b421a3e93c7e9956c55cc92fbe687715e7: Status 404 returned error can't find the container with id 0263b6de560cb4e0afe12fab013438b421a3e93c7e9956c55cc92fbe687715e7 Oct 30 21:56:13.806271 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:13.806143 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2"] Oct 30 21:56:14.316211 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:14.316170 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" event={"ID":"2972d9e0-bf8e-4dc6-b7b8-25caee856f57","Type":"ContainerStarted","Data":"0263b6de560cb4e0afe12fab013438b421a3e93c7e9956c55cc92fbe687715e7"} Oct 30 21:56:14.317512 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:14.317486 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" event={"ID":"1ff003b9-d2ce-4ba0-b05a-5cca5c076893","Type":"ContainerStarted","Data":"33f43272f45d8701791a6aafcb0de2f3daefd699acdf500a670b8355419a858b"} Oct 30 21:56:17.333175 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:17.333144 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" event={"ID":"2972d9e0-bf8e-4dc6-b7b8-25caee856f57","Type":"ContainerStarted","Data":"6e5ab0bb8f67efadc07296b0f688b0d53607fb97c588d4c99c883a40aa15de63"} Oct 30 21:56:17.333633 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:17.333286 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:17.334594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:17.334570 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" event={"ID":"1ff003b9-d2ce-4ba0-b05a-5cca5c076893","Type":"ContainerStarted","Data":"461d1d1885458a4f97197d5e0f36c590128311d3405e9e8ed958d3f6f16a1813"} Oct 30 21:56:17.334713 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:17.334692 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:17.446497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:17.446450 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" podStartSLOduration=2.35504838 podStartE2EDuration="5.446437783s" totalImagesPullingTime="3.091389403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:17.445464618 +0000 UTC m=+556.846020741" watchObservedRunningTime="2026-10-30 21:56:17.446437783 +0000 UTC m=+556.846993903" Oct 30 21:56:17.564147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:17.564096 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" podStartSLOduration=2.657479022 podStartE2EDuration="5.564084017s" totalImagesPullingTime="2.906604995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:17.563381881 +0000 UTC m=+556.963938001" watchObservedRunningTime="2026-10-30 21:56:17.564084017 +0000 UTC m=+556.964640138" Oct 30 21:56:20.299839 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.299807 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:20.649919 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.649848 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh"] Oct 30 21:56:20.653681 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.653665 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.665846 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.665828 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb14762-8c2d-40da-91a6-0141b812fb3c-metrics-certs\") pod \"ai-gateway-controller-9ddfdbfbf-zn9hh\" (UID: \"8eb14762-8c2d-40da-91a6-0141b812fb3c\") " pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.665930 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.665861 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsshn\" (UniqueName: \"kubernetes.io/projected/8eb14762-8c2d-40da-91a6-0141b812fb3c-kube-api-access-xsshn\") pod \"ai-gateway-controller-9ddfdbfbf-zn9hh\" (UID: \"8eb14762-8c2d-40da-91a6-0141b812fb3c\") " pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.708020 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.708000 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh"] Oct 30 21:56:20.766250 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.766227 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb14762-8c2d-40da-91a6-0141b812fb3c-metrics-certs\") pod \"ai-gateway-controller-9ddfdbfbf-zn9hh\" (UID: \"8eb14762-8c2d-40da-91a6-0141b812fb3c\") " pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.766368 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.766263 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsshn\" (UniqueName: \"kubernetes.io/projected/8eb14762-8c2d-40da-91a6-0141b812fb3c-kube-api-access-xsshn\") pod \"ai-gateway-controller-9ddfdbfbf-zn9hh\" (UID: \"8eb14762-8c2d-40da-91a6-0141b812fb3c\") " pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.768891 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.768870 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8eb14762-8c2d-40da-91a6-0141b812fb3c-metrics-certs\") pod \"ai-gateway-controller-9ddfdbfbf-zn9hh\" (UID: \"8eb14762-8c2d-40da-91a6-0141b812fb3c\") " pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.824763 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.824737 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsshn\" (UniqueName: \"kubernetes.io/projected/8eb14762-8c2d-40da-91a6-0141b812fb3c-kube-api-access-xsshn\") pod \"ai-gateway-controller-9ddfdbfbf-zn9hh\" (UID: \"8eb14762-8c2d-40da-91a6-0141b812fb3c\") " pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:20.965686 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:20.965658 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:21.330193 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:21.330168 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh"] Oct 30 21:56:21.332452 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:56:21.332422 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb14762_8c2d_40da_91a6_0141b812fb3c.slice/crio-96976726d3b9e5153433b298d5c692a08c9a0217e6569c93d6bd873940b656e9 WatchSource:0}: Error finding container 96976726d3b9e5153433b298d5c692a08c9a0217e6569c93d6bd873940b656e9: Status 404 returned error can't find the container with id 96976726d3b9e5153433b298d5c692a08c9a0217e6569c93d6bd873940b656e9 Oct 30 21:56:21.352513 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:21.352482 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" event={"ID":"8eb14762-8c2d-40da-91a6-0141b812fb3c","Type":"ContainerStarted","Data":"96976726d3b9e5153433b298d5c692a08c9a0217e6569c93d6bd873940b656e9"} Oct 30 21:56:22.358288 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:22.358259 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" event={"ID":"8eb14762-8c2d-40da-91a6-0141b812fb3c","Type":"ContainerStarted","Data":"876e308d1bf65e6acd67910213b39925f34ba4bdc602f5662c7c2f3e64e9c024"} Oct 30 21:56:22.358641 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:22.358321 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:22.469375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:22.469318 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" podStartSLOduration=1.561486456 podStartE2EDuration="2.469300766s" totalImagesPullingTime="907.81431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:22.468446934 +0000 UTC m=+561.869003055" watchObservedRunningTime="2026-10-30 21:56:22.469300766 +0000 UTC m=+561.869856889" Oct 30 21:56:22.861625 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:56:22.861596 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:56:28.341818 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:28.341791 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-8cgnk" Oct 30 21:56:28.342294 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:28.341862 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9xpg2" Oct 30 21:56:33.366198 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.366166 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-9ddfdbfbf-zn9hh" Oct 30 21:56:33.524056 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.524027 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-bjh6x"] Oct 30 21:56:33.524235 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.524213 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" podUID="3bb95dbc-b350-4263-aa46-26d464d029a5" containerName="manager" containerID="cri-o://45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f" gracePeriod=30 Oct 30 21:56:33.771664 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.771642 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:33.869205 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.869179 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt\" (UniqueName: \"kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt\") pod \"3bb95dbc-b350-4263-aa46-26d464d029a5\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " Oct 30 21:56:33.869339 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.869248 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs\") pod \"3bb95dbc-b350-4263-aa46-26d464d029a5\" (UID: \"3bb95dbc-b350-4263-aa46-26d464d029a5\") " Oct 30 21:56:33.871301 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.871278 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs" pod "3bb95dbc-b350-4263-aa46-26d464d029a5" (UID: "3bb95dbc-b350-4263-aa46-26d464d029a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:33.871390 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.871350 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt" pod "3bb95dbc-b350-4263-aa46-26d464d029a5" (UID: "3bb95dbc-b350-4263-aa46-26d464d029a5"). InnerVolumeSpecName "kube-api-access-7q6qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:33.970255 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.970228 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q6qt\" (UniqueName: \"kubernetes.io/projected/3bb95dbc-b350-4263-aa46-26d464d029a5-kube-api-access-7q6qt\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:33.970255 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:33.970250 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb95dbc-b350-4263-aa46-26d464d029a5-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:34.413987 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.413888 5607 generic.go:356] "Generic (PLEG): container finished" podID="3bb95dbc-b350-4263-aa46-26d464d029a5" containerID="45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f" exitCode=0 Oct 30 21:56:34.413987 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.413940 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" Oct 30 21:56:34.413987 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.413979 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" event={"ID":"3bb95dbc-b350-4263-aa46-26d464d029a5","Type":"ContainerDied","Data":"45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f"} Oct 30 21:56:34.414554 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.414028 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-bjh6x" event={"ID":"3bb95dbc-b350-4263-aa46-26d464d029a5","Type":"ContainerDied","Data":"88b6a263dfd5e1bdaab09452057ca0372e042f3e76c567b153bea19092460250"} Oct 30 21:56:34.414554 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.414053 5607 scope.go:122] "RemoveContainer" containerID="45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f" Oct 30 21:56:34.422978 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.422942 5607 scope.go:122] "RemoveContainer" containerID="45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f" Oct 30 21:56:34.423233 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:56:34.423217 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f\": container with ID starting with 45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f not found: ID does not exist" containerID="45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f" Oct 30 21:56:34.423286 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.423240 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f"} err="failed to get container status \"45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f\": rpc error: code = NotFound desc = could not find container \"45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f\": container with ID starting with 45ecf474dce25f349874478aa5edc5147bdc80c5b07e250f82b7ed43de6e8a4f not found: ID does not exist" Oct 30 21:56:34.568208 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.568183 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-bjh6x"] Oct 30 21:56:34.615486 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.615464 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-bjh6x"] Oct 30 21:56:34.868285 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:34.868250 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb95dbc-b350-4263-aa46-26d464d029a5" path="/var/lib/kubelet/pods/3bb95dbc-b350-4263-aa46-26d464d029a5/volumes" Oct 30 21:56:43.923561 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:43.923525 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn"] Oct 30 21:56:43.924085 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:43.923898 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3bb95dbc-b350-4263-aa46-26d464d029a5" containerName="manager" Oct 30 21:56:43.924085 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:43.923909 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb95dbc-b350-4263-aa46-26d464d029a5" containerName="manager" Oct 30 21:56:43.924085 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:43.924009 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb95dbc-b350-4263-aa46-26d464d029a5" containerName="manager" Oct 30 21:56:43.927062 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:43.927042 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.023320 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.023297 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:56:44.023320 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.023298 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:56:44.023320 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.023306 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 21:56:44.023523 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.023304 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-phhhm\"" Oct 30 21:56:44.025404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.025387 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn"] Oct 30 21:56:44.044286 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.044264 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcsk\" (UniqueName: \"kubernetes.io/projected/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-kube-api-access-lxcsk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.044414 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.044311 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.044414 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.044327 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.044414 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.044396 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.044524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.044483 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.044524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.044503 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145020 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcsk\" (UniqueName: \"kubernetes.io/projected/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-kube-api-access-lxcsk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145183 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145065 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145183 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145083 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145183 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145102 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145183 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145123 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145183 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145140 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145498 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145476 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145593 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145557 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.145593 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.145581 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.147684 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.147657 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.147781 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.147728 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.203262 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.203243 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcsk\" (UniqueName: \"kubernetes.io/projected/ed0dca8e-43d3-4eae-a64e-1638db5fdacf-kube-api-access-lxcsk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn\" (UID: \"ed0dca8e-43d3-4eae-a64e-1638db5fdacf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.240416 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.240392 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:44.611542 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:44.611522 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn"] Oct 30 21:56:44.613633 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:56:44.613604 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0dca8e_43d3_4eae_a64e_1638db5fdacf.slice/crio-8fdf719c5f0c4505fc1f123585a972284fd226478606dba3d736320bacb28ca7 WatchSource:0}: Error finding container 8fdf719c5f0c4505fc1f123585a972284fd226478606dba3d736320bacb28ca7: Status 404 returned error can't find the container with id 8fdf719c5f0c4505fc1f123585a972284fd226478606dba3d736320bacb28ca7 Oct 30 21:56:45.467876 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:45.467836 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" event={"ID":"ed0dca8e-43d3-4eae-a64e-1638db5fdacf","Type":"ContainerStarted","Data":"8fdf719c5f0c4505fc1f123585a972284fd226478606dba3d736320bacb28ca7"} Oct 30 21:56:46.473839 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:46.473804 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" event={"ID":"ed0dca8e-43d3-4eae-a64e-1638db5fdacf","Type":"ContainerStarted","Data":"e3528948e9d91d4632e8125f84b0ce85ef90e84a2f4ec1653c4dab19266bb5d0"} Oct 30 21:56:46.592834 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:46.592776 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" podStartSLOduration=1.85180898 podStartE2EDuration="3.59276181s" totalImagesPullingTime="1.74095283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:46.592180089 +0000 UTC m=+585.992736236" watchObservedRunningTime="2026-10-30 21:56:46.59276181 +0000 UTC m=+585.993317993" Oct 30 21:56:47.241313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:47.241278 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:52.863835 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:56:52.863798 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:56:54.241436 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.241400 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:54.254038 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.254011 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:54.352350 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.352326 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w"] Oct 30 21:56:54.355780 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.355766 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.425292 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.425270 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.425432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.425331 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.425432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.425393 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg64d\" (UniqueName: \"kubernetes.io/projected/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-kube-api-access-mg64d\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.425514 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.425498 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.425552 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.425529 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.425600 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.425561 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.427418 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.427401 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:56:54.428574 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.428556 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w"] Oct 30 21:56:54.525882 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.525824 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn" Oct 30 21:56:54.526045 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526028 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526102 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526054 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg64d\" (UniqueName: \"kubernetes.io/projected/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-kube-api-access-mg64d\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526102 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526078 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526102 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526096 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526247 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526131 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526247 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526160 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526518 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526496 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526590 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526531 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.526590 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.526561 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.528369 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.528350 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.528618 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.528604 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.584226 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.584204 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg64d\" (UniqueName: \"kubernetes.io/projected/a41e3c61-de7d-40bc-9e0a-1874cf3777c1-kube-api-access-mg64d\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b7l5w\" (UID: \"a41e3c61-de7d-40bc-9e0a-1874cf3777c1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:54.671617 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:54.671588 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:55.038274 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:55.038252 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w"] Oct 30 21:56:55.040872 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:56:55.040842 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41e3c61_de7d_40bc_9e0a_1874cf3777c1.slice/crio-7125373c91a2cc739316fcc80af1d31f3113916da647284085708fcd2e814694 WatchSource:0}: Error finding container 7125373c91a2cc739316fcc80af1d31f3113916da647284085708fcd2e814694: Status 404 returned error can't find the container with id 7125373c91a2cc739316fcc80af1d31f3113916da647284085708fcd2e814694 Oct 30 21:56:55.520465 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:55.520434 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" event={"ID":"a41e3c61-de7d-40bc-9e0a-1874cf3777c1","Type":"ContainerStarted","Data":"7125373c91a2cc739316fcc80af1d31f3113916da647284085708fcd2e814694"} Oct 30 21:56:56.527147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:56.527101 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" event={"ID":"a41e3c61-de7d-40bc-9e0a-1874cf3777c1","Type":"ContainerStarted","Data":"8acbfb53fdad776579285007e6d5caa4562db61d72978244cfa2d2eca9adb5db"} Oct 30 21:56:56.639220 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:56.639169 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" podStartSLOduration=2.079358535 podStartE2EDuration="2.639154665s" totalImagesPullingTime="559.79613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:56.638520025 +0000 UTC m=+596.039076145" watchObservedRunningTime="2026-10-30 21:56:56.639154665 +0000 UTC m=+596.039710786" Oct 30 21:56:56.672614 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:56.672590 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:56:57.606453 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.606417 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6"] Oct 30 21:56:57.624194 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.624170 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.665429 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.665404 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6"] Oct 30 21:56:57.710502 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.710467 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:56:57.753197 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.753169 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.753338 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.753205 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.753338 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.753235 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.753410 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.753382 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxqh\" (UniqueName: \"kubernetes.io/projected/635e884d-8183-4f26-b773-cc004f23195b-kube-api-access-vsxqh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.753452 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.753438 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.753482 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.753474 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e884d-8183-4f26-b773-cc004f23195b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854190 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854161 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854316 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854200 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e884d-8183-4f26-b773-cc004f23195b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854316 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854239 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854316 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854254 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854316 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854276 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854316 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854296 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxqh\" (UniqueName: \"kubernetes.io/projected/635e884d-8183-4f26-b773-cc004f23195b-kube-api-access-vsxqh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854619 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854595 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854700 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854622 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.854700 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.854676 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.856641 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.856589 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/635e884d-8183-4f26-b773-cc004f23195b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.856875 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.856860 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635e884d-8183-4f26-b773-cc004f23195b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.912320 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.912295 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxqh\" (UniqueName: \"kubernetes.io/projected/635e884d-8183-4f26-b773-cc004f23195b-kube-api-access-vsxqh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9mgj6\" (UID: \"635e884d-8183-4f26-b773-cc004f23195b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:57.939116 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:57.939091 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:56:58.310727 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:58.310703 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6"] Oct 30 21:56:58.314013 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:56:58.313978 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635e884d_8183_4f26_b773_cc004f23195b.slice/crio-1939384ac52ee4f2c796e2e84be13670b823ecdc39ed57ad8abb21e1874ad776 WatchSource:0}: Error finding container 1939384ac52ee4f2c796e2e84be13670b823ecdc39ed57ad8abb21e1874ad776: Status 404 returned error can't find the container with id 1939384ac52ee4f2c796e2e84be13670b823ecdc39ed57ad8abb21e1874ad776 Oct 30 21:56:58.538152 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:58.538122 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" event={"ID":"635e884d-8183-4f26-b773-cc004f23195b","Type":"ContainerStarted","Data":"1939384ac52ee4f2c796e2e84be13670b823ecdc39ed57ad8abb21e1874ad776"} Oct 30 21:56:59.544432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:59.544400 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" event={"ID":"635e884d-8183-4f26-b773-cc004f23195b","Type":"ContainerStarted","Data":"d5d596ff63266b4396aa203ff06eede040633ce0bda2b7f6d3005e5378e1dcb2"} Oct 30 21:56:59.657734 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:59.657687 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" podStartSLOduration=2.039765472 podStartE2EDuration="2.657671679s" totalImagesPullingTime="617.906207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:59.656367304 +0000 UTC m=+599.056923437" watchObservedRunningTime="2026-10-30 21:56:59.657671679 +0000 UTC m=+599.058227800" Oct 30 21:56:59.939677 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:56:59.939658 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:57:00.928331 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:00.928307 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:57:00.941654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:00.941636 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 21:57:04.118079 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.118039 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xr5st"] Oct 30 21:57:04.122819 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.122796 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.175227 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.175199 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:57:04.203373 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.203340 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f150d8f-d2b5-4904-b322-c19317461e15-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.203502 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.203389 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7686b\" (UniqueName: \"kubernetes.io/projected/6f150d8f-d2b5-4904-b322-c19317461e15-kube-api-access-7686b\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.203543 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.203514 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.203602 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.203580 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.203638 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.203616 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-home\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.203673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.203658 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.204514 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.204495 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xr5st"] Oct 30 21:57:04.304113 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304083 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7686b\" (UniqueName: \"kubernetes.io/projected/6f150d8f-d2b5-4904-b322-c19317461e15-kube-api-access-7686b\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304238 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304171 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304238 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304203 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304238 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304226 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-home\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304405 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304248 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304405 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304336 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f150d8f-d2b5-4904-b322-c19317461e15-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304608 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304588 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304675 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304642 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-home\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.304738 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.304688 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.306496 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.306476 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f150d8f-d2b5-4904-b322-c19317461e15-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.306826 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.306808 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f150d8f-d2b5-4904-b322-c19317461e15-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.362332 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.362301 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7686b\" (UniqueName: \"kubernetes.io/projected/6f150d8f-d2b5-4904-b322-c19317461e15-kube-api-access-7686b\") pod \"echo-simulator-kserve-64b6f8f96c-xr5st\" (UID: \"6f150d8f-d2b5-4904-b322-c19317461e15\") " pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.437051 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.437030 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:04.671912 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.671877 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:57:04.684048 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.684022 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:57:04.812266 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:57:04.812240 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f150d8f_d2b5_4904_b322_c19317461e15.slice/crio-4407f642397654e7c6d94c145963ef713e44dd6012b92f5335734a4b806eba03 WatchSource:0}: Error finding container 4407f642397654e7c6d94c145963ef713e44dd6012b92f5335734a4b806eba03: Status 404 returned error can't find the container with id 4407f642397654e7c6d94c145963ef713e44dd6012b92f5335734a4b806eba03 Oct 30 21:57:04.813661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.813644 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:57:04.844086 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:04.844057 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-xr5st"] Oct 30 21:57:05.573677 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:05.573649 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" event={"ID":"6f150d8f-d2b5-4904-b322-c19317461e15","Type":"ContainerStarted","Data":"4407f642397654e7c6d94c145963ef713e44dd6012b92f5335734a4b806eba03"} Oct 30 21:57:05.587445 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:05.587425 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b7l5w" Oct 30 21:57:07.584770 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:07.584737 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" event={"ID":"6f150d8f-d2b5-4904-b322-c19317461e15","Type":"ContainerStarted","Data":"96a3fe716261325273f223e5facae656887fa3c585118239a7b3fb2bbfb9b755"} Oct 30 21:57:07.696590 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:07.696543 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" podStartSLOduration=1.464302693 podStartE2EDuration="3.696529625s" totalImagesPullingTime="2.232226932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:07.69521594 +0000 UTC m=+607.095772061" watchObservedRunningTime="2026-10-30 21:57:07.696529625 +0000 UTC m=+607.097085746" Oct 30 21:57:07.940132 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:07.940108 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:57:07.952094 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:07.952076 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:57:08.437430 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:08.437407 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:08.600045 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:08.600018 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9mgj6" Oct 30 21:57:10.023854 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.023812 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz"] Oct 30 21:57:10.027648 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.027625 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.077176 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.077152 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:57:10.124132 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.124110 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-twq7g\"" Oct 30 21:57:10.125822 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.125802 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz"] Oct 30 21:57:10.154618 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154592 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.154725 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154620 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5tz\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.154725 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154654 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.154725 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154700 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.154891 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154749 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.154891 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154767 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.154891 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154846 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.155039 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154913 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.155039 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.154948 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256069 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256044 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256184 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256084 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256184 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256105 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256184 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256158 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256183 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5tz\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256271 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256308 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256351 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256555 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256380 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256619 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256597 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256677 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256637 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256806 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256781 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.256891 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.256851 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.257223 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.257199 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.258631 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.258607 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.258937 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.258920 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.314497 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.314434 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.314686 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.314668 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5tz\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz\") pod \"maas-default-gateway-openshift-default-6f54779779-kg4gz\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.340450 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.340432 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:10.800182 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:10.800128 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz"] Oct 30 21:57:10.802326 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:57:10.802300 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110e6a5c_b832_44b6_bf4d_fb91d24fb59a.slice/crio-4d3ff5d82b3e38e00adcab88eb0353207508d17e5c536d8b459efd04bea1bcd6 WatchSource:0}: Error finding container 4d3ff5d82b3e38e00adcab88eb0353207508d17e5c536d8b459efd04bea1bcd6: Status 404 returned error can't find the container with id 4d3ff5d82b3e38e00adcab88eb0353207508d17e5c536d8b459efd04bea1bcd6 Oct 30 21:57:11.605544 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:11.605514 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" event={"ID":"110e6a5c-b832-44b6-bf4d-fb91d24fb59a","Type":"ContainerStarted","Data":"4d3ff5d82b3e38e00adcab88eb0353207508d17e5c536d8b459efd04bea1bcd6"} Oct 30 21:57:14.437710 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:14.437673 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:14.451226 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:14.451199 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:14.634798 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:14.634767 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-xr5st" Oct 30 21:57:15.272389 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:15.272358 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:15.272455 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:15.272428 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:15.272495 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:15.272457 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:15.628566 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:15.628473 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" event={"ID":"110e6a5c-b832-44b6-bf4d-fb91d24fb59a","Type":"ContainerStarted","Data":"457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df"} Oct 30 21:57:15.791633 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:15.791566 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podStartSLOduration=2.324208789 podStartE2EDuration="6.79154896s" totalImagesPullingTime="4.467340171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:15.790883945 +0000 UTC m=+615.191440079" watchObservedRunningTime="2026-10-30 21:57:15.79154896 +0000 UTC m=+615.192105084" Oct 30 21:57:16.341197 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:16.341165 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:16.342682 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:16.342654 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:16.342811 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:16.342707 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:17.341331 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:17.341294 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:17.341708 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:17.341346 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:18.340696 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:18.340669 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:18.340865 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:18.340718 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:19.340853 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:19.340817 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:19.341287 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:19.340870 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:20.341338 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:20.341305 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:20.341788 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:20.341360 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:21.341569 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:21.341533 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:21.341944 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:21.341587 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:22.340936 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:22.340898 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-kg4gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:57:22.341111 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:22.340986 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:57:22.862722 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:57:22.862686 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:57:23.491620 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:23.491587 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:23.492022 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:23.491998 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:23.492770 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:23.492748 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 21:57:40.305483 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.305393 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx"] Oct 30 21:57:40.313334 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.313306 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420500 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420470 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflqb\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420626 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420517 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420626 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420545 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420699 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420635 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420699 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420670 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420764 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420736 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420764 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420759 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420826 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420777 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.420826 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.420807 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.424744 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.424721 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx"] Oct 30 21:57:40.522266 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522241 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522386 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522272 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522386 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522293 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522386 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522311 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522386 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522328 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522386 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522372 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522415 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflqb\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522463 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522657 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522494 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522728 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522784 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522873 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522792 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.522873 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.522835 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.523436 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.523414 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.525160 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.525139 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.525266 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.525242 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.601307 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.601262 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.606630 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.606611 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflqb\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb\") pod \"maas-default-gateway-openshift-default-6f54779779-gqbnx\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:40.627817 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:40.627798 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:41.042552 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.042526 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx"] Oct 30 21:57:41.044849 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:57:41.044821 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4542fc41_1931_4104_b621_086e14239149.slice/crio-c456f9cc9e1eddcae29d0c5d71b4b3acb166011f9df414b5543b78ad6644438f WatchSource:0}: Error finding container c456f9cc9e1eddcae29d0c5d71b4b3acb166011f9df414b5543b78ad6644438f: Status 404 returned error can't find the container with id c456f9cc9e1eddcae29d0c5d71b4b3acb166011f9df414b5543b78ad6644438f Oct 30 21:57:41.046880 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.046849 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:41.047007 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.046916 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:41.047007 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.046947 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:41.750251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.750213 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" event={"ID":"4542fc41-1931-4104-b621-086e14239149","Type":"ContainerStarted","Data":"c18c7b64106f71f1b6a573e4a1bbe49a24ab14e47eeec6316dc1b6a817c38154"} Oct 30 21:57:41.750734 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.750275 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" event={"ID":"4542fc41-1931-4104-b621-086e14239149","Type":"ContainerStarted","Data":"c456f9cc9e1eddcae29d0c5d71b4b3acb166011f9df414b5543b78ad6644438f"} Oct 30 21:57:41.970519 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:41.970462 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podStartSLOduration=1.970445824 podStartE2EDuration="1.970445824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:41.922797302 +0000 UTC m=+641.323353423" watchObservedRunningTime="2026-10-30 21:57:41.970445824 +0000 UTC m=+641.371001947" Oct 30 21:57:42.628623 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:42.628591 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:42.630230 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:42.630205 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:57:42.630359 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:42.630251 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:57:43.628607 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:43.628566 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:57:43.629004 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:43.628627 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:57:44.628901 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:44.628859 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:57:44.629371 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:44.628918 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:57:45.628820 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:45.628785 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:57:45.629013 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:45.628840 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:57:46.628883 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:46.628850 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:57:46.629063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:46.628903 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:57:47.628117 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:47.628082 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:57:47.628299 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:47.628143 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:57:49.208346 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:49.208307 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:49.208843 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:49.208600 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:49.209443 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:57:49.209422 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 21:57:52.862896 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:57:52.862852 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:58:15.257937 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.257902 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78"] Oct 30 21:58:15.261463 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.261445 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.313753 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.313723 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-63d97fec-openshift-default-dockercfg-djfkl\"" Oct 30 21:58:15.363229 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.363196 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78"] Oct 30 21:58:15.409631 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.409603 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.409765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.409634 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.409765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.409663 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.409765 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.409723 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.409930 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.409871 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.410017 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.409941 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.410073 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.410039 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkzp\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.410127 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.410088 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.410164 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.410143 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511475 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511412 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511475 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511459 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkzp\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511483 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511500 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511551 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511569 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511588 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.511673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511613 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.512016 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511866 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.512016 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511863 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.512016 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.511927 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.512162 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.512063 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.512256 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.512236 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.512454 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.512430 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.514067 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.514045 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.514154 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.514067 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.570265 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.570235 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.570367 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.570350 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkzp\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp\") pod \"e2e-ait-63d97fec-openshift-default-65568db4dc-chz78\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:15.577552 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:15.577534 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:16.016267 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:16.016244 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78"] Oct 30 21:58:16.018738 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:16.018708 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf34a24_b2b2_4a6a_ae80_5e80ed6418e1.slice/crio-ac86a3e73956d7f5760d7c00b30ee0d3738076047269d27c624e1d37dc958922 WatchSource:0}: Error finding container ac86a3e73956d7f5760d7c00b30ee0d3738076047269d27c624e1d37dc958922: Status 404 returned error can't find the container with id ac86a3e73956d7f5760d7c00b30ee0d3738076047269d27c624e1d37dc958922 Oct 30 21:58:16.020854 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:16.020821 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:16.020938 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:16.020885 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:16.020938 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:16.020931 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:16.917578 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:16.917545 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" event={"ID":"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1","Type":"ContainerStarted","Data":"81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb"} Oct 30 21:58:16.917970 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:16.917595 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" event={"ID":"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1","Type":"ContainerStarted","Data":"ac86a3e73956d7f5760d7c00b30ee0d3738076047269d27c624e1d37dc958922"} Oct 30 21:58:17.578105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.578072 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:17.583618 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.583596 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:17.699220 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.699170 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podStartSLOduration=2.699155486 podStartE2EDuration="2.699155486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:17.040310684 +0000 UTC m=+676.440866824" watchObservedRunningTime="2026-10-30 21:58:17.699155486 +0000 UTC m=+677.099711606" Oct 30 21:58:17.859009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.858934 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67"] Oct 30 21:58:17.862313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.862291 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.912152 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.912131 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0b0026e9-openshift-default-dockercfg-q2ck4\"" Oct 30 21:58:17.923266 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.923244 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67"] Oct 30 21:58:17.923627 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.923590 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:17.924630 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.924612 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:17.937054 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937035 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937074 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937204 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937161 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937305 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937292 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937342 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937317 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937341 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937360 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937551 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937527 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfs8\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:17.937690 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:17.937675 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039007 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039220 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039057 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039220 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039081 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039346 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039211 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039346 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039248 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039346 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039286 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfs8\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039499 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039357 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039499 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039428 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039499 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039490 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039771 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039745 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039910 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039811 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.039910 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.039848 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.040080 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.040026 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.040266 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.040244 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.041630 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.041612 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.041840 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.041821 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.098374 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.098350 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.099425 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.099402 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfs8\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8\") pod \"e2e-ait-0b0026e9-openshift-default-c8b545777-skq67\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.175780 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.175711 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:18.592270 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.592189 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67"] Oct 30 21:58:18.595039 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:18.595015 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5a059c_434e_4a23_8f56_645a0ef56cc5.slice/crio-a93ef80e076557b5cf97dc2cf6f66e76e253ecfaac3410e0b48948ed096a7d04 WatchSource:0}: Error finding container a93ef80e076557b5cf97dc2cf6f66e76e253ecfaac3410e0b48948ed096a7d04: Status 404 returned error can't find the container with id a93ef80e076557b5cf97dc2cf6f66e76e253ecfaac3410e0b48948ed096a7d04 Oct 30 21:58:18.597624 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.597593 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:18.597689 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.597667 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:18.597724 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.597697 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:18.938609 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.938554 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" event={"ID":"7f5a059c-434e-4a23-8f56-645a0ef56cc5","Type":"ContainerStarted","Data":"18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7"} Oct 30 21:58:18.939033 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:18.938687 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" event={"ID":"7f5a059c-434e-4a23-8f56-645a0ef56cc5","Type":"ContainerStarted","Data":"a93ef80e076557b5cf97dc2cf6f66e76e253ecfaac3410e0b48948ed096a7d04"} Oct 30 21:58:19.058045 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:19.057991 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podStartSLOduration=2.057974671 podStartE2EDuration="2.057974671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:19.057096899 +0000 UTC m=+678.457653020" watchObservedRunningTime="2026-10-30 21:58:19.057974671 +0000 UTC m=+678.458530784" Oct 30 21:58:19.176997 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:19.176972 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:19.182022 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:19.182000 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:19.943302 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:19.943263 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:19.944222 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:19.944201 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:22.775221 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.775191 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt"] Oct 30 21:58:22.779030 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.779014 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:22.828685 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.828661 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2a41d4db\"" Oct 30 21:58:22.875878 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.875849 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2a41d4db-dockercfg-84787\"" Oct 30 21:58:22.878738 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:58:22.878694 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:58:22.878892 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.878718 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt"] Oct 30 21:58:22.882766 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.882738 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qwb\" (UniqueName: \"kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb\") pod \"payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:22.882913 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.882864 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:22.924207 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.924182 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj"] Oct 30 21:58:22.928001 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.927983 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:22.972058 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.972031 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj"] Oct 30 21:58:22.983797 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.983766 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qwb\" (UniqueName: \"kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb\") pod \"payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:22.983890 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.983811 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:22.984573 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:22.984551 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:23.050207 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.050176 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qwb\" (UniqueName: \"kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb\") pod \"payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:23.084365 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.084342 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.084458 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.084377 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pfs\" (UniqueName: \"kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs\") pod \"payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.092510 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.092478 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:23.185518 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.185481 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.185682 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.185548 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pfs\" (UniqueName: \"kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs\") pod \"payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.186997 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.186873 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.243890 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.243860 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pfs\" (UniqueName: \"kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs\") pod \"payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.458073 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.458043 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt"] Oct 30 21:58:23.460574 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:23.460544 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd84c57_7016_4bf8_a580_262a6aab91fd.slice/crio-ac1896976d73d340b480cd590e1c0303667d8a8175e73d64d902c87ec8ba9812 WatchSource:0}: Error finding container ac1896976d73d340b480cd590e1c0303667d8a8175e73d64d902c87ec8ba9812: Status 404 returned error can't find the container with id ac1896976d73d340b480cd590e1c0303667d8a8175e73d64d902c87ec8ba9812 Oct 30 21:58:23.540716 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.540697 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:23.913234 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.913209 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj"] Oct 30 21:58:23.915114 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:23.915084 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac97bb0_552e_44b2_aebe_10373271c517.slice/crio-8f5205ca5903dbb880ae354acc92fd8a979ccc15bbbb247d3c2848d215049e66 WatchSource:0}: Error finding container 8f5205ca5903dbb880ae354acc92fd8a979ccc15bbbb247d3c2848d215049e66: Status 404 returned error can't find the container with id 8f5205ca5903dbb880ae354acc92fd8a979ccc15bbbb247d3c2848d215049e66 Oct 30 21:58:23.962716 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.962686 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" event={"ID":"4bd84c57-7016-4bf8-a580-262a6aab91fd","Type":"ContainerStarted","Data":"a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63"} Oct 30 21:58:23.962823 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.962743 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" event={"ID":"4bd84c57-7016-4bf8-a580-262a6aab91fd","Type":"ContainerStarted","Data":"ac1896976d73d340b480cd590e1c0303667d8a8175e73d64d902c87ec8ba9812"} Oct 30 21:58:23.962923 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.962897 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:23.964304 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:23.964277 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" event={"ID":"dac97bb0-552e-44b2-aebe-10373271c517","Type":"ContainerStarted","Data":"8f5205ca5903dbb880ae354acc92fd8a979ccc15bbbb247d3c2848d215049e66"} Oct 30 21:58:24.073666 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:24.073622 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" podStartSLOduration=2.073609779 podStartE2EDuration="2.073609779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:24.072219909 +0000 UTC m=+683.472776032" watchObservedRunningTime="2026-10-30 21:58:24.073609779 +0000 UTC m=+683.474165899" Oct 30 21:58:24.970409 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:24.970363 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" event={"ID":"dac97bb0-552e-44b2-aebe-10373271c517","Type":"ContainerStarted","Data":"12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd"} Oct 30 21:58:24.970790 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:24.970573 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:25.083779 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:25.083723 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" podStartSLOduration=3.083707814 podStartE2EDuration="3.083707814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:25.082904823 +0000 UTC m=+684.483460945" watchObservedRunningTime="2026-10-30 21:58:25.083707814 +0000 UTC m=+684.484263941" Oct 30 21:58:27.829138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.829103 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv"] Oct 30 21:58:27.832907 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.832891 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:27.887272 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.887250 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv"] Oct 30 21:58:27.932025 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.932007 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4b2e37ae-dockercfg-np9mw\"" Oct 30 21:58:27.932025 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.932024 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4b2e37ae\"" Oct 30 21:58:27.933339 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.933320 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5c4\" (UniqueName: \"kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4\") pod \"payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:27.933387 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.933352 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:27.980455 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.980432 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf"] Oct 30 21:58:27.988286 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:27.988255 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.025916 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.025765 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf"] Oct 30 21:58:28.034823 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.034785 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5c4\" (UniqueName: \"kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4\") pod \"payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:28.035038 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.034843 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:28.036927 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.036900 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:28.094851 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.094797 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5c4\" (UniqueName: \"kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4\") pod \"payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:28.136405 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.136381 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztm2\" (UniqueName: \"kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2\") pod \"payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.136509 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.136452 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.146313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.146291 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:28.237874 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.237837 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.238054 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.238032 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztm2\" (UniqueName: \"kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2\") pod \"payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.238524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.238507 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.296314 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.296290 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztm2\" (UniqueName: \"kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2\") pod \"payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.308150 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.308132 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:28.510669 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.510647 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv"] Oct 30 21:58:28.512529 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:28.512507 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b706356_19d3_4290_90db_d41f3f7d9f02.slice/crio-09ff4190a60b17c2f2a6ef1ebce0cf5e015318e371e65120ef36182b041fa0c8 WatchSource:0}: Error finding container 09ff4190a60b17c2f2a6ef1ebce0cf5e015318e371e65120ef36182b041fa0c8: Status 404 returned error can't find the container with id 09ff4190a60b17c2f2a6ef1ebce0cf5e015318e371e65120ef36182b041fa0c8 Oct 30 21:58:28.673976 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:28.673927 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf"] Oct 30 21:58:28.679154 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:28.679124 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded585a3a_a7f5_4ae0_8ea4_837d6e011bae.slice/crio-f9b26796a71b7ff4c6c4e05d92f830d5064e96015c89fc88cf41058b7d6ae013 WatchSource:0}: Error finding container f9b26796a71b7ff4c6c4e05d92f830d5064e96015c89fc88cf41058b7d6ae013: Status 404 returned error can't find the container with id f9b26796a71b7ff4c6c4e05d92f830d5064e96015c89fc88cf41058b7d6ae013 Oct 30 21:58:29.000378 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.000342 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" event={"ID":"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae","Type":"ContainerStarted","Data":"d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb"} Oct 30 21:58:29.000890 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.000412 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:29.000890 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.000435 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" event={"ID":"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae","Type":"ContainerStarted","Data":"f9b26796a71b7ff4c6c4e05d92f830d5064e96015c89fc88cf41058b7d6ae013"} Oct 30 21:58:29.002038 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.002016 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" event={"ID":"4b706356-19d3-4290-90db-d41f3f7d9f02","Type":"ContainerStarted","Data":"720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044"} Oct 30 21:58:29.002131 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.002055 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" event={"ID":"4b706356-19d3-4290-90db-d41f3f7d9f02","Type":"ContainerStarted","Data":"09ff4190a60b17c2f2a6ef1ebce0cf5e015318e371e65120ef36182b041fa0c8"} Oct 30 21:58:29.002169 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.002142 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:29.111981 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.111922 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" podStartSLOduration=2.111908367 podStartE2EDuration="2.111908367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:29.109701292 +0000 UTC m=+688.510257413" watchObservedRunningTime="2026-10-30 21:58:29.111908367 +0000 UTC m=+688.512464488" Oct 30 21:58:29.222568 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:29.222530 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" podStartSLOduration=2.22251882 podStartE2EDuration="2.22251882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:29.21999533 +0000 UTC m=+688.620551453" watchObservedRunningTime="2026-10-30 21:58:29.22251882 +0000 UTC m=+688.623074940" Oct 30 21:58:30.454935 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.454904 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7"] Oct 30 21:58:30.458727 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.458709 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.508103 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.508077 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-63d97fec\"" Oct 30 21:58:30.556617 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.554840 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-63d97fec-dockercfg-kczlj\"" Oct 30 21:58:30.557424 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.557397 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7"] Oct 30 21:58:30.562171 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.562145 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgfc\" (UniqueName: \"kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc\") pod \"payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.562440 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.562422 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.603187 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.603163 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56"] Oct 30 21:58:30.606709 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.606692 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.653728 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.653701 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56"] Oct 30 21:58:30.663575 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.663540 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thk2b\" (UniqueName: \"kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b\") pod \"payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.663702 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.663610 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgfc\" (UniqueName: \"kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc\") pod \"payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.663758 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.663720 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume\") pod \"payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.663814 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.663777 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.664650 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.664626 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.722825 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.722766 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgfc\" (UniqueName: \"kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc\") pod \"payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.764275 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.764254 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume\") pod \"payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.764386 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.764308 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thk2b\" (UniqueName: \"kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b\") pod \"payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.764899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.764881 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume\") pod \"payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.772095 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.772075 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:30.825764 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.824795 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thk2b\" (UniqueName: \"kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b\") pod \"payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:30.919394 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:30.919362 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:31.139378 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:31.139351 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7"] Oct 30 21:58:31.140745 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:31.140723 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03210bff_5609_41f7_9f1e_0a05146eb2b5.slice/crio-56cb9ff66a0a5fa4dfa3bf86f515f46f10034903788f00e4dc2dfcad4bd5bdf4 WatchSource:0}: Error finding container 56cb9ff66a0a5fa4dfa3bf86f515f46f10034903788f00e4dc2dfcad4bd5bdf4: Status 404 returned error can't find the container with id 56cb9ff66a0a5fa4dfa3bf86f515f46f10034903788f00e4dc2dfcad4bd5bdf4 Oct 30 21:58:31.284484 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:31.284461 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56"] Oct 30 21:58:31.286630 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:31.286602 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef2204a_3c92_41b8_b2ae_5e42c30996ef.slice/crio-b5ddbf3bdda78b0f278df3f7d8637b1a1aaca32044825e12867db865479ae93f WatchSource:0}: Error finding container b5ddbf3bdda78b0f278df3f7d8637b1a1aaca32044825e12867db865479ae93f: Status 404 returned error can't find the container with id b5ddbf3bdda78b0f278df3f7d8637b1a1aaca32044825e12867db865479ae93f Oct 30 21:58:32.022188 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.022151 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" event={"ID":"1ef2204a-3c92-41b8-b2ae-5e42c30996ef","Type":"ContainerStarted","Data":"03cb6b65704ce750f468dc34e3161799d50e184a062185b8f51089efd889c8bb"} Oct 30 21:58:32.022683 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.022242 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" event={"ID":"1ef2204a-3c92-41b8-b2ae-5e42c30996ef","Type":"ContainerStarted","Data":"b5ddbf3bdda78b0f278df3f7d8637b1a1aaca32044825e12867db865479ae93f"} Oct 30 21:58:32.022683 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.022275 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:32.023881 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.023850 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" event={"ID":"03210bff-5609-41f7-9f1e-0a05146eb2b5","Type":"ContainerStarted","Data":"6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc"} Oct 30 21:58:32.024012 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.023907 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:32.024012 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.023931 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" event={"ID":"03210bff-5609-41f7-9f1e-0a05146eb2b5","Type":"ContainerStarted","Data":"56cb9ff66a0a5fa4dfa3bf86f515f46f10034903788f00e4dc2dfcad4bd5bdf4"} Oct 30 21:58:32.133147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.133103 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" podStartSLOduration=2.133091125 podStartE2EDuration="2.133091125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:32.132401686 +0000 UTC m=+691.532957808" watchObservedRunningTime="2026-10-30 21:58:32.133091125 +0000 UTC m=+691.533647246" Oct 30 21:58:32.244120 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:32.244077 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" podStartSLOduration=2.244067315 podStartE2EDuration="2.244067315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:32.241917045 +0000 UTC m=+691.642473178" watchObservedRunningTime="2026-10-30 21:58:32.244067315 +0000 UTC m=+691.644623483" Oct 30 21:58:33.132093 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.132063 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d"] Oct 30 21:58:33.136068 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.136046 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.186148 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.186119 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0b0026e9\"" Oct 30 21:58:33.186275 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.186201 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0b0026e9-dockercfg-q7nms\"" Oct 30 21:58:33.189174 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.189152 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hvt\" (UniqueName: \"kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt\") pod \"payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.189328 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.189312 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.236364 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.236333 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d"] Oct 30 21:58:33.251210 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.251183 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc"] Oct 30 21:58:33.256263 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.256242 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.290453 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.290427 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.290582 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.290462 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.290629 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.290585 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hvt\" (UniqueName: \"kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt\") pod \"payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.290661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.290642 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvd7\" (UniqueName: \"kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7\") pod \"payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.291207 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.291192 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.322006 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.312189 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc"] Oct 30 21:58:33.348944 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.348917 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hvt\" (UniqueName: \"kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt\") pod \"payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.391809 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.391751 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.391936 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.391833 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvd7\" (UniqueName: \"kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7\") pod \"payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.392416 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.392396 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.449873 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.449843 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:33.453513 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.453290 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvd7\" (UniqueName: \"kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7\") pod \"payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.570599 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.570562 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:33.834917 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.834895 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d"] Oct 30 21:58:33.836942 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:33.836910 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0076cb74_75a6_4e03_b450_ae79d7b2aaaa.slice/crio-f4fc1b5b1fa8a9d68e5986b90015c19c448bec32b86c15ae3b509e729acc144e WatchSource:0}: Error finding container f4fc1b5b1fa8a9d68e5986b90015c19c448bec32b86c15ae3b509e729acc144e: Status 404 returned error can't find the container with id f4fc1b5b1fa8a9d68e5986b90015c19c448bec32b86c15ae3b509e729acc144e Oct 30 21:58:33.964520 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:33.964495 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc"] Oct 30 21:58:33.965938 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:33.965910 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b93c855_4469_4ac5_83d5_9a49f21aa747.slice/crio-326607c61b23963e8962384781c60d1be0aa6fc0c71e992304551fd86d2b645e WatchSource:0}: Error finding container 326607c61b23963e8962384781c60d1be0aa6fc0c71e992304551fd86d2b645e: Status 404 returned error can't find the container with id 326607c61b23963e8962384781c60d1be0aa6fc0c71e992304551fd86d2b645e Oct 30 21:58:34.040985 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:34.040937 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" event={"ID":"0076cb74-75a6-4e03-b450-ae79d7b2aaaa","Type":"ContainerStarted","Data":"8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430"} Oct 30 21:58:34.041105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:34.041044 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" event={"ID":"0076cb74-75a6-4e03-b450-ae79d7b2aaaa","Type":"ContainerStarted","Data":"f4fc1b5b1fa8a9d68e5986b90015c19c448bec32b86c15ae3b509e729acc144e"} Oct 30 21:58:34.041105 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:34.041075 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:34.042603 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:34.042578 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" event={"ID":"1b93c855-4469-4ac5-83d5-9a49f21aa747","Type":"ContainerStarted","Data":"326607c61b23963e8962384781c60d1be0aa6fc0c71e992304551fd86d2b645e"} Oct 30 21:58:34.151510 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:34.151453 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" podStartSLOduration=1.151437455 podStartE2EDuration="1.151437455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:34.150841421 +0000 UTC m=+693.551397555" watchObservedRunningTime="2026-10-30 21:58:34.151437455 +0000 UTC m=+693.551993577" Oct 30 21:58:34.973690 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:34.973653 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:58:35.049556 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:35.049518 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" event={"ID":"1b93c855-4469-4ac5-83d5-9a49f21aa747","Type":"ContainerStarted","Data":"f6238656c5bd83990cac9fbd23c5e1615922792c3b496c168f6534484dd76042"} Oct 30 21:58:35.211978 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:35.195205 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" podStartSLOduration=2.195186434 podStartE2EDuration="2.195186434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:35.193127334 +0000 UTC m=+694.593683455" watchObservedRunningTime="2026-10-30 21:58:35.195186434 +0000 UTC m=+694.595742559" Oct 30 21:58:35.978860 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:35.978826 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:58:36.055741 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:36.055712 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:38.287684 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.287658 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n"] Oct 30 21:58:38.291393 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.291378 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.346375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.346352 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n"] Oct 30 21:58:38.368019 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.367990 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp96x\" (UniqueName: \"kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x\") pod \"payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.368197 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.368178 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.390506 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.390476 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4705e99c\"" Oct 30 21:58:38.390729 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.390489 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4705e99c-dockercfg-769x4\"" Oct 30 21:58:38.469265 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.469191 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.469265 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.469243 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp96x\" (UniqueName: \"kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x\") pod \"payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.470006 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.469983 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.528376 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.528348 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp96x\" (UniqueName: \"kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x\") pod \"payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.604884 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.604852 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:38.971228 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:38.971135 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n"] Oct 30 21:58:38.974371 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:58:38.974332 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400bbd66_5849_48c0_8dcc_f152ccb7187d.slice/crio-eb56c35ce71436e747ae5b7d380e215206e2b679f11241c6ff80a78e86bf8ce5 WatchSource:0}: Error finding container eb56c35ce71436e747ae5b7d380e215206e2b679f11241c6ff80a78e86bf8ce5: Status 404 returned error can't find the container with id eb56c35ce71436e747ae5b7d380e215206e2b679f11241c6ff80a78e86bf8ce5 Oct 30 21:58:39.075996 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:39.075944 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" event={"ID":"400bbd66-5849-48c0-8dcc-f152ccb7187d","Type":"ContainerStarted","Data":"a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f"} Oct 30 21:58:39.076229 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:39.076034 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:39.076229 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:39.076059 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" event={"ID":"400bbd66-5849-48c0-8dcc-f152ccb7187d","Type":"ContainerStarted","Data":"eb56c35ce71436e747ae5b7d380e215206e2b679f11241c6ff80a78e86bf8ce5"} Oct 30 21:58:39.187329 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:39.187258 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" podStartSLOduration=1.187239284 podStartE2EDuration="1.187239284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:39.185218832 +0000 UTC m=+698.585774968" watchObservedRunningTime="2026-10-30 21:58:39.187239284 +0000 UTC m=+698.587795407" Oct 30 21:58:40.011024 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:40.010997 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:58:40.011500 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:40.011066 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:58:42.094007 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:42.093976 5607 generic.go:356] "Generic (PLEG): container finished" podID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerID="81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb" exitCode=0 Oct 30 21:58:42.094436 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:42.094057 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" event={"ID":"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1","Type":"ContainerDied","Data":"81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb"} Oct 30 21:58:42.094436 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:42.094389 5607 scope.go:122] "RemoveContainer" containerID="81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb" Oct 30 21:58:42.095585 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:42.095560 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:42.095636 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:42.095627 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:42.095672 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:42.095659 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:43.055264 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:43.055235 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:58:43.055861 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:43.055836 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:58:43.149068 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:43.148738 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" event={"ID":"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1","Type":"ContainerStarted","Data":"b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7"} Oct 30 21:58:43.578614 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:43.578586 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:43.579792 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:43.579772 5607 patch_prober.go:29] interesting pod/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:58:43.579881 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:43.579819 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:58:44.578068 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:44.578034 5607 patch_prober.go:29] interesting pod/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:58:44.578457 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:44.578093 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:58:45.053036 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:45.053009 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:58:45.578403 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:45.578377 5607 patch_prober.go:29] interesting pod/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:58:45.578883 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:45.578858 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:58:46.578983 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:46.578795 5607 patch_prober.go:29] interesting pod/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:58:46.578983 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:46.578857 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:58:47.062780 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:47.062756 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:58:47.578252 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:47.577891 5607 patch_prober.go:29] interesting pod/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:58:47.578252 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:47.577939 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:58:47.923839 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:47.923801 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:48.584109 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:48.584079 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:48.585321 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:48.585297 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:58:50.086567 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:50.086538 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:58:50.985132 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:50.985088 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:58:50.985301 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:50.985171 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:58:51.195720 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:51.195690 5607 generic.go:356] "Generic (PLEG): container finished" podID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerID="18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7" exitCode=0 Oct 30 21:58:51.196202 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:51.195763 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" event={"ID":"7f5a059c-434e-4a23-8f56-645a0ef56cc5","Type":"ContainerDied","Data":"18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7"} Oct 30 21:58:51.196282 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:51.196264 5607 scope.go:122] "RemoveContainer" containerID="18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7" Oct 30 21:58:51.197636 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:51.197604 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:51.197724 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:51.197694 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:51.197762 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:51.197732 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:52.226443 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:52.226406 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" event={"ID":"7f5a059c-434e-4a23-8f56-645a0ef56cc5","Type":"ContainerStarted","Data":"61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028"} Oct 30 21:58:52.867292 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:58:52.867249 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:58:53.176929 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:53.176842 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:53.177123 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:53.177032 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:58:53.177123 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:53.177083 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:58:54.177119 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:54.177085 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:58:54.177609 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:54.177146 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:58:55.176722 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:55.176689 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:58:55.176901 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:55.176744 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:58:56.177031 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:56.176998 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:58:56.177453 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:56.177046 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:58:57.176540 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:57.176504 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:58:57.176755 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:57.176572 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 21:58:58.180296 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:58.180260 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:58.180734 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:58.180689 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:58:58.181314 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:58:58.181295 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:59:08.677524 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.677489 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm"] Oct 30 21:59:08.681173 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.681154 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.733403 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.733358 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:08.737860 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.737832 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.737995 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.737899 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8n65\" (UniqueName: \"kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.737995 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.737940 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.780080 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.780056 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 21:59:08.781850 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.781831 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm"] Oct 30 21:59:08.838527 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.838503 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.838663 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.838646 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.838731 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.838702 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8n65\" (UniqueName: \"kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.839339 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.839315 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.841026 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.841000 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.897214 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.897193 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8n65\" (UniqueName: \"kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65\") pod \"maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:08.993172 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:08.993152 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:09.369769 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:09.369745 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm"] Oct 30 21:59:09.370907 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:09.370875 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee255d2_3677_4a23_88bd_54b66fdbef4f.slice/crio-e30d32d9cdc551369d4c198a1b64a0412d219cef8394cd04506f015dcd2ab20d WatchSource:0}: Error finding container e30d32d9cdc551369d4c198a1b64a0412d219cef8394cd04506f015dcd2ab20d: Status 404 returned error can't find the container with id e30d32d9cdc551369d4c198a1b64a0412d219cef8394cd04506f015dcd2ab20d Oct 30 21:59:10.167081 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:10.167006 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56"] Oct 30 21:59:10.283812 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:10.283774 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7"] Oct 30 21:59:10.328273 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:10.328248 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" event={"ID":"4ee255d2-3677-4a23-88bd-54b66fdbef4f","Type":"ContainerStarted","Data":"e30d32d9cdc551369d4c198a1b64a0412d219cef8394cd04506f015dcd2ab20d"} Oct 30 21:59:14.353069 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:14.352990 5607 generic.go:356] "Generic (PLEG): container finished" podID="4ee255d2-3677-4a23-88bd-54b66fdbef4f" containerID="f9b0f7ae4ee8db6c7e636d9a523a6265c228027e7f51f2bcc09d69ec8367d1cc" exitCode=0 Oct 30 21:59:14.353469 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:14.353087 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" event={"ID":"4ee255d2-3677-4a23-88bd-54b66fdbef4f","Type":"ContainerDied","Data":"f9b0f7ae4ee8db6c7e636d9a523a6265c228027e7f51f2bcc09d69ec8367d1cc"} Oct 30 21:59:15.168036 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.167997 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" podUID="1ef2204a-3c92-41b8-b2ae-5e42c30996ef" containerName="payload-processing" containerID="cri-o://03cb6b65704ce750f468dc34e3161799d50e184a062185b8f51089efd889c8bb" gracePeriod=25 Oct 30 21:59:15.284656 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.284622 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" podUID="03210bff-5609-41f7-9f1e-0a05146eb2b5" containerName="payload-pre-processing" containerID="cri-o://6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc" gracePeriod=25 Oct 30 21:59:15.360117 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.360089 5607 generic.go:356] "Generic (PLEG): container finished" podID="1ef2204a-3c92-41b8-b2ae-5e42c30996ef" containerID="03cb6b65704ce750f468dc34e3161799d50e184a062185b8f51089efd889c8bb" exitCode=0 Oct 30 21:59:15.360466 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.360161 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" event={"ID":"1ef2204a-3c92-41b8-b2ae-5e42c30996ef","Type":"ContainerDied","Data":"03cb6b65704ce750f468dc34e3161799d50e184a062185b8f51089efd889c8bb"} Oct 30 21:59:15.438677 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.438276 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:59:15.500149 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.500121 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume\") pod \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " Oct 30 21:59:15.500276 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.500177 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b\" (UniqueName: \"kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b\") pod \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\" (UID: \"1ef2204a-3c92-41b8-b2ae-5e42c30996ef\") " Oct 30 21:59:15.500557 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.500524 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume" pod "1ef2204a-3c92-41b8-b2ae-5e42c30996ef" (UID: "1ef2204a-3c92-41b8-b2ae-5e42c30996ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:15.500756 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.500725 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:15.502720 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.502699 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b" pod "1ef2204a-3c92-41b8-b2ae-5e42c30996ef" (UID: "1ef2204a-3c92-41b8-b2ae-5e42c30996ef"). InnerVolumeSpecName "kube-api-access-thk2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:15.509017 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.508985 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:15.553889 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.553862 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:59:15.601781 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.601753 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc\" (UniqueName: \"kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc\") pod \"03210bff-5609-41f7-9f1e-0a05146eb2b5\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " Oct 30 21:59:15.601993 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.601830 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca\") pod \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " Oct 30 21:59:15.601993 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.601909 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65\" (UniqueName: \"kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65\") pod \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " Oct 30 21:59:15.601993 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.601975 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca\") pod \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\" (UID: \"4ee255d2-3677-4a23-88bd-54b66fdbef4f\") " Oct 30 21:59:15.602178 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.602052 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume\") pod \"03210bff-5609-41f7-9f1e-0a05146eb2b5\" (UID: \"03210bff-5609-41f7-9f1e-0a05146eb2b5\") " Oct 30 21:59:15.602365 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.602340 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thk2b\" (UniqueName: \"kubernetes.io/projected/1ef2204a-3c92-41b8-b2ae-5e42c30996ef-kube-api-access-thk2b\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:15.602879 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.602568 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca" pod "4ee255d2-3677-4a23-88bd-54b66fdbef4f" (UID: "4ee255d2-3677-4a23-88bd-54b66fdbef4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:15.602879 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.602655 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume" pod "03210bff-5609-41f7-9f1e-0a05146eb2b5" (UID: "03210bff-5609-41f7-9f1e-0a05146eb2b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:15.604715 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.604693 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc" pod "03210bff-5609-41f7-9f1e-0a05146eb2b5" (UID: "03210bff-5609-41f7-9f1e-0a05146eb2b5"). InnerVolumeSpecName "kube-api-access-rrgfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:15.604828 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.604729 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65" pod "4ee255d2-3677-4a23-88bd-54b66fdbef4f" (UID: "4ee255d2-3677-4a23-88bd-54b66fdbef4f"). InnerVolumeSpecName "kube-api-access-q8n65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:15.604828 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.604792 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca" pod "4ee255d2-3677-4a23-88bd-54b66fdbef4f" (UID: "4ee255d2-3677-4a23-88bd-54b66fdbef4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:15.702852 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.702826 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8n65\" (UniqueName: \"kubernetes.io/projected/4ee255d2-3677-4a23-88bd-54b66fdbef4f-kube-api-access-q8n65\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:15.702852 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.702851 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:15.703040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.702861 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03210bff-5609-41f7-9f1e-0a05146eb2b5-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:15.703040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.702870 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrgfc\" (UniqueName: \"kubernetes.io/projected/03210bff-5609-41f7-9f1e-0a05146eb2b5-kube-api-access-rrgfc\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:15.703040 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:15.702880 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee255d2-3677-4a23-88bd-54b66fdbef4f-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.301760 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.301732 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j"] Oct 30 21:59:16.302241 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302228 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4ee255d2-3677-4a23-88bd-54b66fdbef4f" containerName="revoke-keys" Oct 30 21:59:16.302299 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302242 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee255d2-3677-4a23-88bd-54b66fdbef4f" containerName="revoke-keys" Oct 30 21:59:16.302299 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302257 5607 cpu_manager.go:414] "containerMap: removing container" podUID="03210bff-5609-41f7-9f1e-0a05146eb2b5" containerName="payload-pre-processing" Oct 30 21:59:16.302299 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302264 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03210bff-5609-41f7-9f1e-0a05146eb2b5" containerName="payload-pre-processing" Oct 30 21:59:16.302299 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302293 5607 cpu_manager.go:414] "containerMap: removing container" podUID="1ef2204a-3c92-41b8-b2ae-5e42c30996ef" containerName="payload-processing" Oct 30 21:59:16.302299 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302300 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef2204a-3c92-41b8-b2ae-5e42c30996ef" containerName="payload-processing" Oct 30 21:59:16.302476 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302391 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee255d2-3677-4a23-88bd-54b66fdbef4f" containerName="revoke-keys" Oct 30 21:59:16.302476 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302404 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="03210bff-5609-41f7-9f1e-0a05146eb2b5" containerName="payload-pre-processing" Oct 30 21:59:16.302476 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.302415 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef2204a-3c92-41b8-b2ae-5e42c30996ef" containerName="payload-processing" Oct 30 21:59:16.306228 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.306205 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.360604 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.360548 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j"] Oct 30 21:59:16.367485 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.367460 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" Oct 30 21:59:16.367637 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.367460 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56" event={"ID":"1ef2204a-3c92-41b8-b2ae-5e42c30996ef","Type":"ContainerDied","Data":"b5ddbf3bdda78b0f278df3f7d8637b1a1aaca32044825e12867db865479ae93f"} Oct 30 21:59:16.367712 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.367637 5607 scope.go:122] "RemoveContainer" containerID="03cb6b65704ce750f468dc34e3161799d50e184a062185b8f51089efd889c8bb" Oct 30 21:59:16.369375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.369352 5607 generic.go:356] "Generic (PLEG): container finished" podID="03210bff-5609-41f7-9f1e-0a05146eb2b5" containerID="6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc" exitCode=0 Oct 30 21:59:16.369474 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.369407 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" event={"ID":"03210bff-5609-41f7-9f1e-0a05146eb2b5","Type":"ContainerDied","Data":"6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc"} Oct 30 21:59:16.369539 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.369454 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" event={"ID":"03210bff-5609-41f7-9f1e-0a05146eb2b5","Type":"ContainerDied","Data":"56cb9ff66a0a5fa4dfa3bf86f515f46f10034903788f00e4dc2dfcad4bd5bdf4"} Oct 30 21:59:16.369589 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.369553 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7" Oct 30 21:59:16.372215 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.372196 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" Oct 30 21:59:16.372330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.372224 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm" event={"ID":"4ee255d2-3677-4a23-88bd-54b66fdbef4f","Type":"ContainerDied","Data":"e30d32d9cdc551369d4c198a1b64a0412d219cef8394cd04506f015dcd2ab20d"} Oct 30 21:59:16.372330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.372264 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30d32d9cdc551369d4c198a1b64a0412d219cef8394cd04506f015dcd2ab20d" Oct 30 21:59:16.381821 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.381713 5607 scope.go:122] "RemoveContainer" containerID="6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc" Oct 30 21:59:16.391187 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.391170 5607 scope.go:122] "RemoveContainer" containerID="6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc" Oct 30 21:59:16.391486 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:16.391466 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc\": container with ID starting with 6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc not found: ID does not exist" containerID="6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc" Oct 30 21:59:16.391531 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.391497 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc"} err="failed to get container status \"6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc\": rpc error: code = NotFound desc = could not find container \"6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc\": container with ID starting with 6ec8a12144928c149ad8636666d8b3c9cef1ba79d9c3366507caf95c343984dc not found: ID does not exist" Oct 30 21:59:16.409966 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.409922 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.410061 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.409984 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhbv\" (UniqueName: \"kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.410211 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.410191 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.511286 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.511261 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.511387 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.511356 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.511433 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.511392 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhbv\" (UniqueName: \"kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.511999 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.511977 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.513573 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.513552 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.526157 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.526136 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7"] Oct 30 21:59:16.529873 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.529853 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d97fec-6c64cd86f5-l8zw7"] Oct 30 21:59:16.569694 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.569672 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhbv\" (UniqueName: \"kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv\") pod \"maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.623060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.623041 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:16.714126 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.714096 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56"] Oct 30 21:59:16.719646 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.718917 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d97fec-55b7b8d65-bsp56"] Oct 30 21:59:16.869113 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.869034 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03210bff-5609-41f7-9f1e-0a05146eb2b5" path="/var/lib/kubelet/pods/03210bff-5609-41f7-9f1e-0a05146eb2b5/volumes" Oct 30 21:59:16.869507 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.869487 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef2204a-3c92-41b8-b2ae-5e42c30996ef" path="/var/lib/kubelet/pods/1ef2204a-3c92-41b8-b2ae-5e42c30996ef/volumes" Oct 30 21:59:16.992773 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:16.992750 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j"] Oct 30 21:59:16.994190 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:16.994168 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377bb790_5dc9_4f58_8ec4_691f542a8c0f.slice/crio-16052afca4bec56c30897da3fb501fe187672173f65d786999855f4dbc56ee1c WatchSource:0}: Error finding container 16052afca4bec56c30897da3fb501fe187672173f65d786999855f4dbc56ee1c: Status 404 returned error can't find the container with id 16052afca4bec56c30897da3fb501fe187672173f65d786999855f4dbc56ee1c Oct 30 21:59:17.379605 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:17.379571 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" event={"ID":"377bb790-5dc9-4f58-8ec4-691f542a8c0f","Type":"ContainerStarted","Data":"687abd9cc7806a2bd6ab6b862819ce36948b2c43385da7c885b98b942223eb6e"} Oct 30 21:59:17.380018 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:17.379633 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" event={"ID":"377bb790-5dc9-4f58-8ec4-691f542a8c0f","Type":"ContainerStarted","Data":"16052afca4bec56c30897da3fb501fe187672173f65d786999855f4dbc56ee1c"} Oct 30 21:59:17.490315 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:17.490222 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" podStartSLOduration=1.490208806 podStartE2EDuration="1.490208806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:17.489349908 +0000 UTC m=+736.889906035" watchObservedRunningTime="2026-10-30 21:59:17.490208806 +0000 UTC m=+736.890764975" Oct 30 21:59:18.386845 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:18.386810 5607 generic.go:356] "Generic (PLEG): container finished" podID="377bb790-5dc9-4f58-8ec4-691f542a8c0f" containerID="687abd9cc7806a2bd6ab6b862819ce36948b2c43385da7c885b98b942223eb6e" exitCode=0 Oct 30 21:59:18.387287 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:18.386886 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" event={"ID":"377bb790-5dc9-4f58-8ec4-691f542a8c0f","Type":"ContainerDied","Data":"687abd9cc7806a2bd6ab6b862819ce36948b2c43385da7c885b98b942223eb6e"} Oct 30 21:59:19.244302 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.244267 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc"] Oct 30 21:59:19.360467 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.360438 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d"] Oct 30 21:59:19.536215 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.536194 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:19.646565 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.646533 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca\") pod \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " Oct 30 21:59:19.646748 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.646678 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca\") pod \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " Oct 30 21:59:19.646748 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.646716 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv\" (UniqueName: \"kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv\") pod \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\" (UID: \"377bb790-5dc9-4f58-8ec4-691f542a8c0f\") " Oct 30 21:59:19.647115 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.647085 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca" pod "377bb790-5dc9-4f58-8ec4-691f542a8c0f" (UID: "377bb790-5dc9-4f58-8ec4-691f542a8c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:19.648795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.648770 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca" pod "377bb790-5dc9-4f58-8ec4-691f542a8c0f" (UID: "377bb790-5dc9-4f58-8ec4-691f542a8c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:19.649011 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.648988 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv" pod "377bb790-5dc9-4f58-8ec4-691f542a8c0f" (UID: "377bb790-5dc9-4f58-8ec4-691f542a8c0f"). InnerVolumeSpecName "kube-api-access-gfhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:19.747842 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.747807 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:19.747842 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.747844 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhbv\" (UniqueName: \"kubernetes.io/projected/377bb790-5dc9-4f58-8ec4-691f542a8c0f-kube-api-access-gfhbv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:19.748057 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:19.747859 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377bb790-5dc9-4f58-8ec4-691f542a8c0f-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:20.122743 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:20.122710 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78"] Oct 30 21:59:20.123027 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:20.122973 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" containerID="cri-o://b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7" gracePeriod=30 Oct 30 21:59:20.399251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:20.399164 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" event={"ID":"377bb790-5dc9-4f58-8ec4-691f542a8c0f","Type":"ContainerDied","Data":"16052afca4bec56c30897da3fb501fe187672173f65d786999855f4dbc56ee1c"} Oct 30 21:59:20.399251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:20.399190 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j" Oct 30 21:59:20.399251 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:20.399219 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16052afca4bec56c30897da3fb501fe187672173f65d786999855f4dbc56ee1c" Oct 30 21:59:21.572979 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.572939 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv"] Oct 30 21:59:21.573446 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.573432 5607 cpu_manager.go:414] "containerMap: removing container" podUID="377bb790-5dc9-4f58-8ec4-691f542a8c0f" containerName="revoke-keys" Oct 30 21:59:21.573491 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.573447 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377bb790-5dc9-4f58-8ec4-691f542a8c0f" containerName="revoke-keys" Oct 30 21:59:21.573560 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.573551 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="377bb790-5dc9-4f58-8ec4-691f542a8c0f" containerName="revoke-keys" Oct 30 21:59:21.577162 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.577145 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.628649 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.628623 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7f66ab-openshift-default-dockercfg-pk2pq\"" Oct 30 21:59:21.666945 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.666915 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667150 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.666979 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667150 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667052 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667150 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667111 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667343 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667154 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vxmv\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667343 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667182 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667343 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667203 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667343 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667221 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.667343 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.667298 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.678021 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.677996 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv"] Oct 30 21:59:21.768476 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768441 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768492 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768527 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768563 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768610 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vxmv\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768641 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768965 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768669 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768965 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768697 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768965 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768753 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.768965 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.768893 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.769240 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.769012 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.769240 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.769153 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.769240 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.769157 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.769387 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.769364 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.771414 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.771384 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.771520 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.771444 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.828500 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.828438 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.828749 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.828726 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vxmv\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.872570 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.872541 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf"] Oct 30 21:59:21.891432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.891407 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:21.977155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:21.977131 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv"] Oct 30 21:59:22.316991 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.316645 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv"] Oct 30 21:59:22.317974 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:22.317926 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60599210_aca4_4ef9_947a_3b21c31b55d9.slice/crio-ce362eeda1e787f6c5cfdab0f5f2d201971363e5269b1cab73454b7b2bf039b3 WatchSource:0}: Error finding container ce362eeda1e787f6c5cfdab0f5f2d201971363e5269b1cab73454b7b2bf039b3: Status 404 returned error can't find the container with id ce362eeda1e787f6c5cfdab0f5f2d201971363e5269b1cab73454b7b2bf039b3 Oct 30 21:59:22.320876 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.320834 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:22.321012 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.320942 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:22.321078 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.321013 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:22.412330 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.412303 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" event={"ID":"60599210-aca4-4ef9-947a-3b21c31b55d9","Type":"ContainerStarted","Data":"3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7"} Oct 30 21:59:22.412442 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.412357 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" event={"ID":"60599210-aca4-4ef9-947a-3b21c31b55d9","Type":"ContainerStarted","Data":"ce362eeda1e787f6c5cfdab0f5f2d201971363e5269b1cab73454b7b2bf039b3"} Oct 30 21:59:22.530003 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.529931 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podStartSLOduration=1.529917555 podStartE2EDuration="1.529917555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:22.527989124 +0000 UTC m=+741.928545309" watchObservedRunningTime="2026-10-30 21:59:22.529917555 +0000 UTC m=+741.930473729" Oct 30 21:59:22.862833 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:22.862798 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:59:22.891473 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:22.891448 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:23.896667 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:23.896637 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:24.245346 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.245313 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" podUID="1b93c855-4469-4ac5-83d5-9a49f21aa747" containerName="payload-processing" containerID="cri-o://f6238656c5bd83990cac9fbd23c5e1615922792c3b496c168f6534484dd76042" gracePeriod=25 Oct 30 21:59:24.361194 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.361152 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" podUID="0076cb74-75a6-4e03-b450-ae79d7b2aaaa" containerName="payload-pre-processing" containerID="cri-o://8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430" gracePeriod=25 Oct 30 21:59:24.436468 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.436433 5607 generic.go:356] "Generic (PLEG): container finished" podID="1b93c855-4469-4ac5-83d5-9a49f21aa747" containerID="f6238656c5bd83990cac9fbd23c5e1615922792c3b496c168f6534484dd76042" exitCode=0 Oct 30 21:59:24.436617 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.436502 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" event={"ID":"1b93c855-4469-4ac5-83d5-9a49f21aa747","Type":"ContainerDied","Data":"f6238656c5bd83990cac9fbd23c5e1615922792c3b496c168f6534484dd76042"} Oct 30 21:59:24.437009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.436986 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:24.438412 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.438392 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 21:59:24.514229 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.513450 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:59:24.597414 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.597114 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7\" (UniqueName: \"kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7\") pod \"1b93c855-4469-4ac5-83d5-9a49f21aa747\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " Oct 30 21:59:24.597694 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.597672 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume\") pod \"1b93c855-4469-4ac5-83d5-9a49f21aa747\" (UID: \"1b93c855-4469-4ac5-83d5-9a49f21aa747\") " Oct 30 21:59:24.598119 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.598089 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume" pod "1b93c855-4469-4ac5-83d5-9a49f21aa747" (UID: "1b93c855-4469-4ac5-83d5-9a49f21aa747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:24.607456 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.607425 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7" pod "1b93c855-4469-4ac5-83d5-9a49f21aa747" (UID: "1b93c855-4469-4ac5-83d5-9a49f21aa747"). InnerVolumeSpecName "kube-api-access-ztvd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:24.628491 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.628470 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:59:24.700060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.699359 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume\") pod \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " Oct 30 21:59:24.700060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.699432 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt\" (UniqueName: \"kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt\") pod \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\" (UID: \"0076cb74-75a6-4e03-b450-ae79d7b2aaaa\") " Oct 30 21:59:24.700060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.699783 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume" pod "0076cb74-75a6-4e03-b450-ae79d7b2aaaa" (UID: "0076cb74-75a6-4e03-b450-ae79d7b2aaaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:24.700060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.699972 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvd7\" (UniqueName: \"kubernetes.io/projected/1b93c855-4469-4ac5-83d5-9a49f21aa747-kube-api-access-ztvd7\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:24.700060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.699999 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b93c855-4469-4ac5-83d5-9a49f21aa747-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:24.700060 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.700015 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:24.702100 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.702067 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt" pod "0076cb74-75a6-4e03-b450-ae79d7b2aaaa" (UID: "0076cb74-75a6-4e03-b450-ae79d7b2aaaa"). InnerVolumeSpecName "kube-api-access-97hvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:24.801298 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:24.801220 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97hvt\" (UniqueName: \"kubernetes.io/projected/0076cb74-75a6-4e03-b450-ae79d7b2aaaa-kube-api-access-97hvt\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.400337 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.400284 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:59:25.442687 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.442664 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" Oct 30 21:59:25.442820 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.442691 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc" event={"ID":"1b93c855-4469-4ac5-83d5-9a49f21aa747","Type":"ContainerDied","Data":"326607c61b23963e8962384781c60d1be0aa6fc0c71e992304551fd86d2b645e"} Oct 30 21:59:25.442820 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.442782 5607 scope.go:122] "RemoveContainer" containerID="f6238656c5bd83990cac9fbd23c5e1615922792c3b496c168f6534484dd76042" Oct 30 21:59:25.444596 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.444564 5607 generic.go:356] "Generic (PLEG): container finished" podID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerID="b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7" exitCode=0 Oct 30 21:59:25.444721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.444628 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" Oct 30 21:59:25.444791 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.444732 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" event={"ID":"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1","Type":"ContainerDied","Data":"b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7"} Oct 30 21:59:25.444791 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.444777 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78" event={"ID":"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1","Type":"ContainerDied","Data":"ac86a3e73956d7f5760d7c00b30ee0d3738076047269d27c624e1d37dc958922"} Oct 30 21:59:25.446243 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.446223 5607 generic.go:356] "Generic (PLEG): container finished" podID="0076cb74-75a6-4e03-b450-ae79d7b2aaaa" containerID="8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430" exitCode=0 Oct 30 21:59:25.446368 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.446252 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" event={"ID":"0076cb74-75a6-4e03-b450-ae79d7b2aaaa","Type":"ContainerDied","Data":"8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430"} Oct 30 21:59:25.446368 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.446274 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" Oct 30 21:59:25.446368 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.446294 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d" event={"ID":"0076cb74-75a6-4e03-b450-ae79d7b2aaaa","Type":"ContainerDied","Data":"f4fc1b5b1fa8a9d68e5986b90015c19c448bec32b86c15ae3b509e729acc144e"} Oct 30 21:59:25.452652 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.452636 5607 scope.go:122] "RemoveContainer" containerID="b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7" Oct 30 21:59:25.460770 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.460750 5607 scope.go:122] "RemoveContainer" containerID="81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb" Oct 30 21:59:25.468337 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.468320 5607 scope.go:122] "RemoveContainer" containerID="b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7" Oct 30 21:59:25.468573 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:25.468556 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7\": container with ID starting with b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7 not found: ID does not exist" containerID="b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7" Oct 30 21:59:25.468618 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.468581 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7"} err="failed to get container status \"b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7\": rpc error: code = NotFound desc = could not find container \"b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7\": container with ID starting with b4cfb9639f29bab8ccd437c82009443550a6ff5fc81529d5fbf318071bb47ad7 not found: ID does not exist" Oct 30 21:59:25.468618 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.468596 5607 scope.go:122] "RemoveContainer" containerID="81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb" Oct 30 21:59:25.468821 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:25.468807 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb\": container with ID starting with 81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb not found: ID does not exist" containerID="81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb" Oct 30 21:59:25.468860 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.468824 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb"} err="failed to get container status \"81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb\": rpc error: code = NotFound desc = could not find container \"81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb\": container with ID starting with 81dff54c9dd0165af96948c43c4980a91fe1560b43fa05d5635a26432617a1cb not found: ID does not exist" Oct 30 21:59:25.468860 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.468836 5607 scope.go:122] "RemoveContainer" containerID="8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430" Oct 30 21:59:25.476360 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.476341 5607 scope.go:122] "RemoveContainer" containerID="8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430" Oct 30 21:59:25.476597 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:25.476579 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430\": container with ID starting with 8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430 not found: ID does not exist" containerID="8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430" Oct 30 21:59:25.476649 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.476601 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430"} err="failed to get container status \"8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430\": rpc error: code = NotFound desc = could not find container \"8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430\": container with ID starting with 8adcc48ad62975c3f033768fd19edef1aa52320900d339e503c90104948b7430 not found: ID does not exist" Oct 30 21:59:25.508223 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508193 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508247 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508278 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508297 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508331 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508403 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508424 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508446 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508509 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo\") pod \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\" (UID: \"dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1\") " Oct 30 21:59:25.508673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508538 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:25.508876 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508855 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:25.508997 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.508972 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:25.509061 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509034 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:25.509435 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509407 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:25.509603 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509584 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.509692 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509604 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.509692 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509620 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.509692 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509635 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.509692 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.509648 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.510877 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.510850 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:25.510877 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.510863 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "kube-api-access-hlkzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:25.511432 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.511404 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:25.511529 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.511429 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token" pod "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" (UID: "dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:25.610505 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.610479 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.610632 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.610508 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlkzp\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-kube-api-access-hlkzp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.610632 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.610526 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.610632 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.610541 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:25.738856 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.738823 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d"] Oct 30 21:59:25.748499 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.748469 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0b0026e9-68f444d9bc-tdp8d"] Oct 30 21:59:25.854857 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.854828 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc"] Oct 30 21:59:25.859475 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.859451 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b0026e9-547c9d965f-w9fzc"] Oct 30 21:59:25.965637 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.965609 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78"] Oct 30 21:59:25.969162 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:25.969132 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-63d97fec-openshift-default-65568db4dc-chz78"] Oct 30 21:59:26.794531 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.794503 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm"] Oct 30 21:59:26.841340 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.841315 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0026e9-4wzzm"] Oct 30 21:59:26.869700 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.869675 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0076cb74-75a6-4e03-b450-ae79d7b2aaaa" path="/var/lib/kubelet/pods/0076cb74-75a6-4e03-b450-ae79d7b2aaaa/volumes" Oct 30 21:59:26.870055 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.870043 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b93c855-4469-4ac5-83d5-9a49f21aa747" path="/var/lib/kubelet/pods/1b93c855-4469-4ac5-83d5-9a49f21aa747/volumes" Oct 30 21:59:26.870404 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.870393 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee255d2-3677-4a23-88bd-54b66fdbef4f" path="/var/lib/kubelet/pods/4ee255d2-3677-4a23-88bd-54b66fdbef4f/volumes" Oct 30 21:59:26.871096 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.871083 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" path="/var/lib/kubelet/pods/dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1/volumes" Oct 30 21:59:26.873000 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.872981 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" podUID="ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" containerName="payload-processing" containerID="cri-o://d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb" gracePeriod=25 Oct 30 21:59:26.978195 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:26.978172 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" podUID="4b706356-19d3-4290-90db-d41f3f7d9f02" containerName="payload-pre-processing" containerID="cri-o://720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044" gracePeriod=25 Oct 30 21:59:27.134434 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.134408 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:59:27.227154 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.227128 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2\" (UniqueName: \"kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2\") pod \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " Oct 30 21:59:27.227278 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.227227 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume\") pod \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\" (UID: \"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae\") " Oct 30 21:59:27.227550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.227524 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume" pod "ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" (UID: "ed585a3a-a7f5-4ae0-8ea4-837d6e011bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:27.229473 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.229448 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2" pod "ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" (UID: "ed585a3a-a7f5-4ae0-8ea4-837d6e011bae"). InnerVolumeSpecName "kube-api-access-8ztm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:27.238942 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.238922 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:59:27.328877 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.328845 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4\" (UniqueName: \"kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4\") pod \"4b706356-19d3-4290-90db-d41f3f7d9f02\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " Oct 30 21:59:27.329050 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.328903 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume\") pod \"4b706356-19d3-4290-90db-d41f3f7d9f02\" (UID: \"4b706356-19d3-4290-90db-d41f3f7d9f02\") " Oct 30 21:59:27.329147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.329135 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ztm2\" (UniqueName: \"kubernetes.io/projected/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-kube-api-access-8ztm2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.329204 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.329149 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.329304 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.329283 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume" pod "4b706356-19d3-4290-90db-d41f3f7d9f02" (UID: "4b706356-19d3-4290-90db-d41f3f7d9f02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:27.331152 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.331132 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4" pod "4b706356-19d3-4290-90db-d41f3f7d9f02" (UID: "4b706356-19d3-4290-90db-d41f3f7d9f02"). InnerVolumeSpecName "kube-api-access-tb5c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:27.430478 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.430396 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5c4\" (UniqueName: \"kubernetes.io/projected/4b706356-19d3-4290-90db-d41f3f7d9f02-kube-api-access-tb5c4\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.430478 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.430421 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b706356-19d3-4290-90db-d41f3f7d9f02-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:27.455190 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.455162 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67"] Oct 30 21:59:27.455394 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.455371 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" containerID="cri-o://61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028" gracePeriod=30 Oct 30 21:59:27.462257 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.462232 5607 generic.go:356] "Generic (PLEG): container finished" podID="ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" containerID="d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb" exitCode=0 Oct 30 21:59:27.462389 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.462270 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" Oct 30 21:59:27.462389 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.462305 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" event={"ID":"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae","Type":"ContainerDied","Data":"d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb"} Oct 30 21:59:27.462389 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.462351 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf" event={"ID":"ed585a3a-a7f5-4ae0-8ea4-837d6e011bae","Type":"ContainerDied","Data":"f9b26796a71b7ff4c6c4e05d92f830d5064e96015c89fc88cf41058b7d6ae013"} Oct 30 21:59:27.462514 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.462393 5607 scope.go:122] "RemoveContainer" containerID="d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb" Oct 30 21:59:27.463905 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.463873 5607 generic.go:356] "Generic (PLEG): container finished" podID="4b706356-19d3-4290-90db-d41f3f7d9f02" containerID="720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044" exitCode=0 Oct 30 21:59:27.464035 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.463918 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" Oct 30 21:59:27.464035 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.463932 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" event={"ID":"4b706356-19d3-4290-90db-d41f3f7d9f02","Type":"ContainerDied","Data":"720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044"} Oct 30 21:59:27.464145 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.464063 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv" event={"ID":"4b706356-19d3-4290-90db-d41f3f7d9f02","Type":"ContainerDied","Data":"09ff4190a60b17c2f2a6ef1ebce0cf5e015318e371e65120ef36182b041fa0c8"} Oct 30 21:59:27.471260 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.471242 5607 scope.go:122] "RemoveContainer" containerID="d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb" Oct 30 21:59:27.471538 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:27.471520 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb\": container with ID starting with d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb not found: ID does not exist" containerID="d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb" Oct 30 21:59:27.471606 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.471545 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb"} err="failed to get container status \"d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb\": rpc error: code = NotFound desc = could not find container \"d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb\": container with ID starting with d1c86091b97e1a796184c1730012d0b6f92001cb81e1c5a5d5e0d28172ba3ffb not found: ID does not exist" Oct 30 21:59:27.471606 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.471562 5607 scope.go:122] "RemoveContainer" containerID="720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044" Oct 30 21:59:27.479612 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.479598 5607 scope.go:122] "RemoveContainer" containerID="720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044" Oct 30 21:59:27.479856 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:27.479836 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044\": container with ID starting with 720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044 not found: ID does not exist" containerID="720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044" Oct 30 21:59:27.479923 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.479861 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044"} err="failed to get container status \"720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044\": rpc error: code = NotFound desc = could not find container \"720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044\": container with ID starting with 720f5e01a32e6ab7c952d0f39e1a730c9aa573641c50f5d36994187ac4b25044 not found: ID does not exist" Oct 30 21:59:27.675963 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.675919 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv"] Oct 30 21:59:27.683372 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.683311 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b2e37ae-57684d7fcf-fqssv"] Oct 30 21:59:27.790548 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.790522 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf"] Oct 30 21:59:27.792015 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:27.791996 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b2e37ae-b99c48959-27bkf"] Oct 30 21:59:28.182576 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.182541 5607 patch_prober.go:29] interesting pod/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:59:28.182967 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.182593 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:59:28.446173 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.446145 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj"] Oct 30 21:59:28.571101 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.571072 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt"] Oct 30 21:59:28.868643 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.868557 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b706356-19d3-4290-90db-d41f3f7d9f02" path="/var/lib/kubelet/pods/4b706356-19d3-4290-90db-d41f3f7d9f02/volumes" Oct 30 21:59:28.869045 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.869024 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" path="/var/lib/kubelet/pods/ed585a3a-a7f5-4ae0-8ea4-837d6e011bae/volumes" Oct 30 21:59:28.908368 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908337 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd"] Oct 30 21:59:28.908792 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908780 5607 cpu_manager.go:414] "containerMap: removing container" podUID="1b93c855-4469-4ac5-83d5-9a49f21aa747" containerName="payload-processing" Oct 30 21:59:28.908832 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908796 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b93c855-4469-4ac5-83d5-9a49f21aa747" containerName="payload-processing" Oct 30 21:59:28.908832 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908821 5607 cpu_manager.go:414] "containerMap: removing container" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" Oct 30 21:59:28.908832 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908827 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908847 5607 cpu_manager.go:414] "containerMap: removing container" podUID="0076cb74-75a6-4e03-b450-ae79d7b2aaaa" containerName="payload-pre-processing" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908853 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0076cb74-75a6-4e03-b450-ae79d7b2aaaa" containerName="payload-pre-processing" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908859 5607 cpu_manager.go:414] "containerMap: removing container" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908864 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908873 5607 cpu_manager.go:414] "containerMap: removing container" podUID="ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" containerName="payload-processing" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908880 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" containerName="payload-processing" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908889 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4b706356-19d3-4290-90db-d41f3f7d9f02" containerName="payload-pre-processing" Oct 30 21:59:28.908920 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908893 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b706356-19d3-4290-90db-d41f3f7d9f02" containerName="payload-pre-processing" Oct 30 21:59:28.909155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908979 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" Oct 30 21:59:28.909155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908990 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="0076cb74-75a6-4e03-b450-ae79d7b2aaaa" containerName="payload-pre-processing" Oct 30 21:59:28.909155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.908996 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf34a24-b2b2-4a6a-ae80-5e80ed6418e1" containerName="istio-proxy" Oct 30 21:59:28.909155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.909002 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed585a3a-a7f5-4ae0-8ea4-837d6e011bae" containerName="payload-processing" Oct 30 21:59:28.909155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.909006 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b706356-19d3-4290-90db-d41f3f7d9f02" containerName="payload-pre-processing" Oct 30 21:59:28.909155 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.909014 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b93c855-4469-4ac5-83d5-9a49f21aa747" containerName="payload-processing" Oct 30 21:59:28.913416 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.913399 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:28.963261 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.963242 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d09fff-openshift-default-dockercfg-596tc\"" Oct 30 21:59:28.969989 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:28.969946 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd"] Oct 30 21:59:29.046755 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.046723 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.046885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.046775 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.046885 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.046867 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.046990 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.046922 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.047034 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.047002 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.047069 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.047041 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7ls\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.047128 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.047109 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.047160 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.047140 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.047204 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.047169 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148365 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148284 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148365 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148339 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148376 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148427 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148450 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148488 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7ls\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148526 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148594 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148547 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148873 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148598 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148931 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148885 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.148931 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.148915 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.149082 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.149011 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.149147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.149121 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.149503 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.149479 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.151067 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.151045 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.151220 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.151201 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.216782 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.216755 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.217135 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.216900 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7ls\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.226729 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.226705 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:29.588617 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.588591 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm"] Oct 30 21:59:29.593321 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.593304 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.644437 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.644413 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd"] Oct 30 21:59:29.645792 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:29.645768 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e271101_b9ba_4b33_8862_88373435f991.slice/crio-98fec42216ee0ecf11909abe13e865c37590bf9ffa7dd044768134d024ce743d WatchSource:0}: Error finding container 98fec42216ee0ecf11909abe13e865c37590bf9ffa7dd044768134d024ce743d: Status 404 returned error can't find the container with id 98fec42216ee0ecf11909abe13e865c37590bf9ffa7dd044768134d024ce743d Oct 30 21:59:29.647945 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.647914 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:29.648067 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.648003 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:29.648067 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.648031 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:29.653930 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.653905 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654062 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.653947 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654062 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.653989 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654062 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.654019 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654062 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.654034 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654189 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.654076 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654189 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.654108 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654189 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.654124 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.654189 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.654151 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvsmp\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.658946 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.658923 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm"] Oct 30 21:59:29.675918 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.675896 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b47e8d-openshift-default-dockercfg-bxx6t\"" Oct 30 21:59:29.755308 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755271 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755308 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755319 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755580 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755344 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755580 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755393 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755580 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755416 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755580 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755490 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755792 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755667 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755792 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755709 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.755792 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.755755 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvsmp\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.756228 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.756191 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.756348 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.756268 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.756480 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.756457 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.756574 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.756527 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.756781 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.756762 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.758136 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.758113 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.759403 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.759381 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.813679 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.813646 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.814559 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.814540 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvsmp\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:29.908332 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:29.908260 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:30.325984 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.325940 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm"] Oct 30 21:59:30.328779 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:30.328745 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1511ebe9_bdff_44e5_a614_9e70273e7e33.slice/crio-4b87e026145677642814ae87e227d9336a912801429550483aa9916381d0d771 WatchSource:0}: Error finding container 4b87e026145677642814ae87e227d9336a912801429550483aa9916381d0d771: Status 404 returned error can't find the container with id 4b87e026145677642814ae87e227d9336a912801429550483aa9916381d0d771 Oct 30 21:59:30.332289 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.332254 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:30.332398 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.332318 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:30.332398 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.332350 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:30.484391 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.484355 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" event={"ID":"1511ebe9-bdff-44e5-a614-9e70273e7e33","Type":"ContainerStarted","Data":"e0dc21d5b3a684ef53948b055fadc035aba7a8b727687531262a4b826fbb82b4"} Oct 30 21:59:30.484586 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.484413 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" event={"ID":"1511ebe9-bdff-44e5-a614-9e70273e7e33","Type":"ContainerStarted","Data":"4b87e026145677642814ae87e227d9336a912801429550483aa9916381d0d771"} Oct 30 21:59:30.486534 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.486503 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" event={"ID":"2e271101-b9ba-4b33-8862-88373435f991","Type":"ContainerStarted","Data":"3fd5e1fc3c708d32f91e95624f4d27977d20e57650809759074725a1aaab4b1e"} Oct 30 21:59:30.486642 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.486557 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" event={"ID":"2e271101-b9ba-4b33-8862-88373435f991","Type":"ContainerStarted","Data":"98fec42216ee0ecf11909abe13e865c37590bf9ffa7dd044768134d024ce743d"} Oct 30 21:59:30.599434 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.599343 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podStartSLOduration=1.599326895 podStartE2EDuration="1.599326895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.598603655 +0000 UTC m=+749.999159777" watchObservedRunningTime="2026-10-30 21:59:30.599326895 +0000 UTC m=+749.999883016" Oct 30 21:59:30.715324 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.715277 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podStartSLOduration=2.715264024 podStartE2EDuration="2.715264024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.713809994 +0000 UTC m=+750.114366117" watchObservedRunningTime="2026-10-30 21:59:30.715264024 +0000 UTC m=+750.115820145" Oct 30 21:59:30.909149 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:30.909067 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:31.226900 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:31.226875 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:31.232636 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:31.232616 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:31.491877 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:31.491799 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:31.492490 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:31.492471 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 21:59:31.914004 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:31.913922 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:32.497167 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.497140 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:32.498063 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.498044 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 21:59:32.724370 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.724315 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:59:32.788465 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788441 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788610 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788497 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788610 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788527 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788695 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788609 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788695 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788646 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788695 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788677 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788840 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788721 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788840 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788780 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.788840 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788828 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket\") pod \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\" (UID: \"7f5a059c-434e-4a23-8f56-645a0ef56cc5\") " Oct 30 21:59:32.789007 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788890 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:32.789007 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.788942 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:32.789108 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789087 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:32.789152 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789100 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:32.789397 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789375 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.789482 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789409 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.789482 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789419 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.789482 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789428 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.789482 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.789379 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:32.790771 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.790742 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:59:32.791169 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.791150 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:32.791420 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.791397 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "kube-api-access-bvfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:32.791420 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.791401 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy" pod "7f5a059c-434e-4a23-8f56-645a0ef56cc5" (UID: "7f5a059c-434e-4a23-8f56-645a0ef56cc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:32.890366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.890329 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.890366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.890365 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a059c-434e-4a23-8f56-645a0ef56cc5-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.890366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.890378 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.890545 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.890387 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvfs8\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-kube-api-access-bvfs8\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.890545 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.890395 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a059c-434e-4a23-8f56-645a0ef56cc5-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:32.938713 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.938689 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h"] Oct 30 21:59:32.939131 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.939118 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" Oct 30 21:59:32.939181 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.939132 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" Oct 30 21:59:32.939226 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.939217 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" Oct 30 21:59:32.939257 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.939231 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" Oct 30 21:59:32.942530 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.942515 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:32.994820 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.994751 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-7f66ab\"" Oct 30 21:59:32.997219 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:32.997196 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h"] Oct 30 21:59:33.092679 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.092649 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wdq\" (UniqueName: \"kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq\") pod \"payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.092863 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.092767 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.125795 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.125763 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg"] Oct 30 21:59:33.126303 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.126287 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" Oct 30 21:59:33.126366 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.126306 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerName="istio-proxy" Oct 30 21:59:33.129596 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.129581 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.184042 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.184013 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-7f66ab-dockercfg-ljr8z\"" Oct 30 21:59:33.184845 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.184825 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg"] Oct 30 21:59:33.193482 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.193458 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wdq\" (UniqueName: \"kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq\") pod \"payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.193619 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.193492 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.193619 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.193529 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhsp\" (UniqueName: \"kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp\") pod \"payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.193736 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.193632 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.194654 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.194628 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.252329 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.252276 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wdq\" (UniqueName: \"kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq\") pod \"payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.255146 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.255128 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:33.295375 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.295345 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.295520 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.295403 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhsp\" (UniqueName: \"kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp\") pod \"payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.296308 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.296281 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.353474 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.353454 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhsp\" (UniqueName: \"kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp\") pod \"payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.441943 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.441916 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:33.447149 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.447121 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" podUID="dac97bb0-552e-44b2-aebe-10373271c517" containerName="payload-processing" containerID="cri-o://12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd" gracePeriod=25 Oct 30 21:59:33.508602 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:33.508525 5607 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac97bb0_552e_44b2_aebe_10373271c517.slice/crio-conmon-12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:59:33.509328 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.509295 5607 generic.go:356] "Generic (PLEG): container finished" podID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" containerID="61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028" exitCode=0 Oct 30 21:59:33.509604 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.509571 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" event={"ID":"7f5a059c-434e-4a23-8f56-645a0ef56cc5","Type":"ContainerDied","Data":"61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028"} Oct 30 21:59:33.509682 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.509634 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" event={"ID":"7f5a059c-434e-4a23-8f56-645a0ef56cc5","Type":"ContainerDied","Data":"a93ef80e076557b5cf97dc2cf6f66e76e253ecfaac3410e0b48948ed096a7d04"} Oct 30 21:59:33.509937 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.509918 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67" Oct 30 21:59:33.510281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.510263 5607 scope.go:122] "RemoveContainer" containerID="61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028" Oct 30 21:59:33.559652 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.559628 5607 scope.go:122] "RemoveContainer" containerID="18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7" Oct 30 21:59:33.568684 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.568667 5607 scope.go:122] "RemoveContainer" containerID="61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028" Oct 30 21:59:33.569000 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:33.568978 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028\": container with ID starting with 61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028 not found: ID does not exist" containerID="61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028" Oct 30 21:59:33.569088 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.569008 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028"} err="failed to get container status \"61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028\": rpc error: code = NotFound desc = could not find container \"61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028\": container with ID starting with 61afd3b652f94213b6c45322f6dfd45bc50d9254ff227c5f76f6cebd07ca0028 not found: ID does not exist" Oct 30 21:59:33.569088 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.569027 5607 scope.go:122] "RemoveContainer" containerID="18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7" Oct 30 21:59:33.569318 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:33.569300 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7\": container with ID starting with 18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7 not found: ID does not exist" containerID="18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7" Oct 30 21:59:33.569360 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.569324 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7"} err="failed to get container status \"18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7\": rpc error: code = NotFound desc = could not find container \"18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7\": container with ID starting with 18139493258b2b00ee25b03e902d29c568cacabec6488f58cac9241e74efe8b7 not found: ID does not exist" Oct 30 21:59:33.571445 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.571403 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" podUID="4bd84c57-7016-4bf8-a580-262a6aab91fd" containerName="payload-pre-processing" containerID="cri-o://a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63" gracePeriod=25 Oct 30 21:59:33.624147 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.624123 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h"] Oct 30 21:59:33.658861 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:33.658816 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd9a3b5_8766_45f2_a471_68f8443cc2f6.slice/crio-73e786502c560372467d52c0584c0ed2a3863b82070655e8418a13eccc26e625 WatchSource:0}: Error finding container 73e786502c560372467d52c0584c0ed2a3863b82070655e8418a13eccc26e625: Status 404 returned error can't find the container with id 73e786502c560372467d52c0584c0ed2a3863b82070655e8418a13eccc26e625 Oct 30 21:59:33.670186 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.670142 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67"] Oct 30 21:59:33.679233 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.679205 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0b0026e9-openshift-default-c8b545777-skq67"] Oct 30 21:59:33.745583 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.745562 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:59:33.813993 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.813967 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg"] Oct 30 21:59:33.816533 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:33.816509 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa9bead_f06b_4467_8d59_fa25bf5354ca.slice/crio-94e4b62304328713a6d26d8c35be270c933cfedffc1936d327f68e7f467d3b8c WatchSource:0}: Error finding container 94e4b62304328713a6d26d8c35be270c933cfedffc1936d327f68e7f467d3b8c: Status 404 returned error can't find the container with id 94e4b62304328713a6d26d8c35be270c933cfedffc1936d327f68e7f467d3b8c Oct 30 21:59:33.839693 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.839670 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:59:33.902516 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.902494 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume\") pod \"dac97bb0-552e-44b2-aebe-10373271c517\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " Oct 30 21:59:33.902803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.902618 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs\" (UniqueName: \"kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs\") pod \"dac97bb0-552e-44b2-aebe-10373271c517\" (UID: \"dac97bb0-552e-44b2-aebe-10373271c517\") " Oct 30 21:59:33.902869 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.902854 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume" pod "dac97bb0-552e-44b2-aebe-10373271c517" (UID: "dac97bb0-552e-44b2-aebe-10373271c517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:33.903039 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.903024 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac97bb0-552e-44b2-aebe-10373271c517-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.904551 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:33.904532 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs" pod "dac97bb0-552e-44b2-aebe-10373271c517" (UID: "dac97bb0-552e-44b2-aebe-10373271c517"). InnerVolumeSpecName "kube-api-access-m7pfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:34.003988 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.003966 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb\" (UniqueName: \"kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb\") pod \"4bd84c57-7016-4bf8-a580-262a6aab91fd\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " Oct 30 21:59:34.004104 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.004006 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume\") pod \"4bd84c57-7016-4bf8-a580-262a6aab91fd\" (UID: \"4bd84c57-7016-4bf8-a580-262a6aab91fd\") " Oct 30 21:59:34.004188 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.004176 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pfs\" (UniqueName: \"kubernetes.io/projected/dac97bb0-552e-44b2-aebe-10373271c517-kube-api-access-m7pfs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:34.004361 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.004341 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume" pod "4bd84c57-7016-4bf8-a580-262a6aab91fd" (UID: "4bd84c57-7016-4bf8-a580-262a6aab91fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:34.006009 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.005986 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb" pod "4bd84c57-7016-4bf8-a580-262a6aab91fd" (UID: "4bd84c57-7016-4bf8-a580-262a6aab91fd"). InnerVolumeSpecName "kube-api-access-n7qwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:34.104889 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.104832 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qwb\" (UniqueName: \"kubernetes.io/projected/4bd84c57-7016-4bf8-a580-262a6aab91fd-kube-api-access-n7qwb\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:34.104889 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.104853 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd84c57-7016-4bf8-a580-262a6aab91fd-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:34.516258 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.516227 5607 generic.go:356] "Generic (PLEG): container finished" podID="4bd84c57-7016-4bf8-a580-262a6aab91fd" containerID="a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63" exitCode=0 Oct 30 21:59:34.516661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.516271 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" Oct 30 21:59:34.516661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.516310 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" event={"ID":"4bd84c57-7016-4bf8-a580-262a6aab91fd","Type":"ContainerDied","Data":"a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63"} Oct 30 21:59:34.516661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.516369 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt" event={"ID":"4bd84c57-7016-4bf8-a580-262a6aab91fd","Type":"ContainerDied","Data":"ac1896976d73d340b480cd590e1c0303667d8a8175e73d64d902c87ec8ba9812"} Oct 30 21:59:34.516661 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.516403 5607 scope.go:122] "RemoveContainer" containerID="a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63" Oct 30 21:59:34.518197 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.518171 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" event={"ID":"bfa9bead-f06b-4467-8d59-fa25bf5354ca","Type":"ContainerStarted","Data":"fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3"} Oct 30 21:59:34.518290 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.518223 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" event={"ID":"bfa9bead-f06b-4467-8d59-fa25bf5354ca","Type":"ContainerStarted","Data":"94e4b62304328713a6d26d8c35be270c933cfedffc1936d327f68e7f467d3b8c"} Oct 30 21:59:34.518751 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.518733 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:34.521869 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.521797 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" event={"ID":"afd9a3b5-8766-45f2-a471-68f8443cc2f6","Type":"ContainerStarted","Data":"e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2"} Oct 30 21:59:34.521869 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.521840 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" event={"ID":"afd9a3b5-8766-45f2-a471-68f8443cc2f6","Type":"ContainerStarted","Data":"73e786502c560372467d52c0584c0ed2a3863b82070655e8418a13eccc26e625"} Oct 30 21:59:34.522050 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.521917 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:34.523308 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.523286 5607 generic.go:356] "Generic (PLEG): container finished" podID="dac97bb0-552e-44b2-aebe-10373271c517" containerID="12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd" exitCode=0 Oct 30 21:59:34.523398 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.523321 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" Oct 30 21:59:34.523398 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.523348 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" event={"ID":"dac97bb0-552e-44b2-aebe-10373271c517","Type":"ContainerDied","Data":"12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd"} Oct 30 21:59:34.523398 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.523382 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj" event={"ID":"dac97bb0-552e-44b2-aebe-10373271c517","Type":"ContainerDied","Data":"8f5205ca5903dbb880ae354acc92fd8a979ccc15bbbb247d3c2848d215049e66"} Oct 30 21:59:34.527399 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.527381 5607 scope.go:122] "RemoveContainer" containerID="a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63" Oct 30 21:59:34.527715 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:34.527695 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63\": container with ID starting with a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63 not found: ID does not exist" containerID="a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63" Oct 30 21:59:34.527773 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.527725 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63"} err="failed to get container status \"a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63\": rpc error: code = NotFound desc = could not find container \"a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63\": container with ID starting with a73c45c75cf0b046f552e95c7660525ffb7857463bda433b1852b9ec16f3da63 not found: ID does not exist" Oct 30 21:59:34.527773 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.527748 5607 scope.go:122] "RemoveContainer" containerID="12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd" Oct 30 21:59:34.538205 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.538183 5607 scope.go:122] "RemoveContainer" containerID="12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd" Oct 30 21:59:34.538478 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:34.538460 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd\": container with ID starting with 12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd not found: ID does not exist" containerID="12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd" Oct 30 21:59:34.538527 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.538484 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd"} err="failed to get container status \"12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd\": rpc error: code = NotFound desc = could not find container \"12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd\": container with ID starting with 12b419ee3065fe5bac624ef9341732da638c82aadb7d46cc97eb431ab469adcd not found: ID does not exist" Oct 30 21:59:34.682868 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.682816 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" podStartSLOduration=2.682800013 podStartE2EDuration="2.682800013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:34.682334726 +0000 UTC m=+754.082890848" watchObservedRunningTime="2026-10-30 21:59:34.682800013 +0000 UTC m=+754.083356140" Oct 30 21:59:34.796509 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.796429 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" podStartSLOduration=1.796416569 podStartE2EDuration="1.796416569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:34.79480127 +0000 UTC m=+754.195357392" watchObservedRunningTime="2026-10-30 21:59:34.796416569 +0000 UTC m=+754.196972690" Oct 30 21:59:34.869313 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.869284 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5a059c-434e-4a23-8f56-645a0ef56cc5" path="/var/lib/kubelet/pods/7f5a059c-434e-4a23-8f56-645a0ef56cc5/volumes" Oct 30 21:59:34.904161 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.904138 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj"] Oct 30 21:59:34.905720 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:34.905698 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a41d4db-564597649f-cb7sj"] Oct 30 21:59:35.010471 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:35.010442 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt"] Oct 30 21:59:35.012445 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:35.012423 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a41d4db-85d5d4dcff-8ttbt"] Oct 30 21:59:36.869138 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:36.869098 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd84c57-7016-4bf8-a580-262a6aab91fd" path="/var/lib/kubelet/pods/4bd84c57-7016-4bf8-a580-262a6aab91fd/volumes" Oct 30 21:59:36.869758 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:36.869615 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac97bb0-552e-44b2-aebe-10373271c517" path="/var/lib/kubelet/pods/dac97bb0-552e-44b2-aebe-10373271c517/volumes" Oct 30 21:59:39.385387 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385356 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh"] Oct 30 21:59:39.385793 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385780 5607 cpu_manager.go:414] "containerMap: removing container" podUID="dac97bb0-552e-44b2-aebe-10373271c517" containerName="payload-processing" Oct 30 21:59:39.385838 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385794 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac97bb0-552e-44b2-aebe-10373271c517" containerName="payload-processing" Oct 30 21:59:39.385838 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385823 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4bd84c57-7016-4bf8-a580-262a6aab91fd" containerName="payload-pre-processing" Oct 30 21:59:39.385838 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385828 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd84c57-7016-4bf8-a580-262a6aab91fd" containerName="payload-pre-processing" Oct 30 21:59:39.385938 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385900 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac97bb0-552e-44b2-aebe-10373271c517" containerName="payload-processing" Oct 30 21:59:39.385938 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.385913 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd84c57-7016-4bf8-a580-262a6aab91fd" containerName="payload-pre-processing" Oct 30 21:59:39.390486 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.390470 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.444727 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.444703 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh"] Oct 30 21:59:39.489601 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.489573 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b47e8d\"" Oct 30 21:59:39.489819 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.489703 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b47e8d-dockercfg-pls4h\"" Oct 30 21:59:39.553055 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.553025 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq"] Oct 30 21:59:39.553635 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.553608 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkfw\" (UniqueName: \"kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw\") pod \"payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.553699 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.553684 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.557042 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.557027 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.586478 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.586452 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq"] Oct 30 21:59:39.654550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.654496 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkfw\" (UniqueName: \"kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw\") pod \"payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.654550 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.654535 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.654700 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.654567 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h46\" (UniqueName: \"kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46\") pod \"payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.654700 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.654625 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.655281 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.655264 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.713114 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.713084 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkfw\" (UniqueName: \"kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw\") pod \"payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:39.755211 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.755175 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h46\" (UniqueName: \"kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46\") pod \"payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.755320 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.755295 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.755890 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.755873 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.813932 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.813908 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h46\" (UniqueName: \"kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46\") pod \"payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:39.870058 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:39.870038 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:40.003074 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.003045 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:40.235173 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.235151 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq"] Oct 30 21:59:40.237081 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:40.237054 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549f9160_eeb0_40e9_81f8_dfeecc639d62.slice/crio-f892ccb7794676795bd830b42c8949c4b47d8d9e2447338916990858a9e30c7b WatchSource:0}: Error finding container f892ccb7794676795bd830b42c8949c4b47d8d9e2447338916990858a9e30c7b: Status 404 returned error can't find the container with id f892ccb7794676795bd830b42c8949c4b47d8d9e2447338916990858a9e30c7b Oct 30 21:59:40.367840 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.367819 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh"] Oct 30 21:59:40.369885 ip-10-0-133-91 kubenswrapper[5607]: W1030 21:59:40.369862 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0b9648_ca39_4146_a501_040489a211c2.slice/crio-26e67150ed7077d0b3b380efa117d022c4893961d7a6aa04de84e3f5cea7a96c WatchSource:0}: Error finding container 26e67150ed7077d0b3b380efa117d022c4893961d7a6aa04de84e3f5cea7a96c: Status 404 returned error can't find the container with id 26e67150ed7077d0b3b380efa117d022c4893961d7a6aa04de84e3f5cea7a96c Oct 30 21:59:40.558348 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.558262 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" event={"ID":"549f9160-eeb0-40e9-81f8-dfeecc639d62","Type":"ContainerStarted","Data":"f9789cc442295ab0a6691509ad05dc1dc80503f2275239b42cfc561aa77eb123"} Oct 30 21:59:40.558348 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.558312 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" event={"ID":"549f9160-eeb0-40e9-81f8-dfeecc639d62","Type":"ContainerStarted","Data":"f892ccb7794676795bd830b42c8949c4b47d8d9e2447338916990858a9e30c7b"} Oct 30 21:59:40.558712 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.558355 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:40.559987 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.559963 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" event={"ID":"2f0b9648-ca39-4146-a501-040489a211c2","Type":"ContainerStarted","Data":"0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2"} Oct 30 21:59:40.560092 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.560009 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" event={"ID":"2f0b9648-ca39-4146-a501-040489a211c2","Type":"ContainerStarted","Data":"26e67150ed7077d0b3b380efa117d022c4893961d7a6aa04de84e3f5cea7a96c"} Oct 30 21:59:40.560092 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.560054 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:40.685721 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.685677 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" podStartSLOduration=1.685663361 podStartE2EDuration="1.685663361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:40.684589522 +0000 UTC m=+760.085145834" watchObservedRunningTime="2026-10-30 21:59:40.685663361 +0000 UTC m=+760.086219483" Oct 30 21:59:40.795607 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:40.795563 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" podStartSLOduration=1.79555479 podStartE2EDuration="1.79555479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:40.794801016 +0000 UTC m=+760.195357154" watchObservedRunningTime="2026-10-30 21:59:40.79555479 +0000 UTC m=+760.196110911" Oct 30 21:59:42.018597 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.018565 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j"] Oct 30 21:59:42.022381 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.022365 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.118665 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.118635 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d09fff-dockercfg-bzvnh\"" Oct 30 21:59:42.120444 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.120426 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j"] Oct 30 21:59:42.138893 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.138871 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d09fff\"" Oct 30 21:59:42.180424 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.180386 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ptz\" (UniqueName: \"kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz\") pod \"payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.180609 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.180551 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.281716 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.281647 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.281716 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.281709 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ptz\" (UniqueName: \"kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz\") pod \"payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.282361 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.282343 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.340062 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.340038 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ptz\" (UniqueName: \"kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz\") pod \"payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.635308 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.635244 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:42.917681 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:42.917614 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n"] Oct 30 21:59:43.026107 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:43.026086 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j"] Oct 30 21:59:43.586232 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:43.586195 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" event={"ID":"d4a033be-69cb-42cc-b8aa-833addc63ed7","Type":"ContainerStarted","Data":"51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44"} Oct 30 21:59:43.586409 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:43.586259 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 21:59:43.586409 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:43.586283 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" event={"ID":"d4a033be-69cb-42cc-b8aa-833addc63ed7","Type":"ContainerStarted","Data":"ecff125edba22728d6f11d1869069cf9f32e6d06c5f72e9dbb0d349d3efab1c9"} Oct 30 21:59:43.697593 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:43.697539 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" podStartSLOduration=2.697526475 podStartE2EDuration="2.697526475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:43.696373102 +0000 UTC m=+763.096929246" watchObservedRunningTime="2026-10-30 21:59:43.697526475 +0000 UTC m=+763.098082596" Oct 30 21:59:45.533478 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:45.533452 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 21:59:46.539541 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:46.539517 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 21:59:47.907277 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:47.907244 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j"] Oct 30 21:59:47.918426 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:47.918359 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" podUID="400bbd66-5849-48c0-8dcc-f152ccb7187d" containerName="payload-pre-processing" containerID="cri-o://a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f" gracePeriod=25 Oct 30 21:59:47.953725 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:47.953704 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a41d4db-9xm2j"] Oct 30 21:59:48.179803 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.179748 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:59:48.339246 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.339220 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume\") pod \"400bbd66-5849-48c0-8dcc-f152ccb7187d\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " Oct 30 21:59:48.339379 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.339359 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x\" (UniqueName: \"kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x\") pod \"400bbd66-5849-48c0-8dcc-f152ccb7187d\" (UID: \"400bbd66-5849-48c0-8dcc-f152ccb7187d\") " Oct 30 21:59:48.339673 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.339647 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume" pod "400bbd66-5849-48c0-8dcc-f152ccb7187d" (UID: "400bbd66-5849-48c0-8dcc-f152ccb7187d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:48.341690 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.341665 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x" pod "400bbd66-5849-48c0-8dcc-f152ccb7187d" (UID: "400bbd66-5849-48c0-8dcc-f152ccb7187d"). InnerVolumeSpecName "kube-api-access-qp96x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:48.440899 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.440880 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp96x\" (UniqueName: \"kubernetes.io/projected/400bbd66-5849-48c0-8dcc-f152ccb7187d-kube-api-access-qp96x\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:48.441015 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.440901 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400bbd66-5849-48c0-8dcc-f152ccb7187d-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:48.615268 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.615245 5607 generic.go:356] "Generic (PLEG): container finished" podID="400bbd66-5849-48c0-8dcc-f152ccb7187d" containerID="a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f" exitCode=0 Oct 30 21:59:48.615399 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.615301 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" Oct 30 21:59:48.615399 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.615339 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" event={"ID":"400bbd66-5849-48c0-8dcc-f152ccb7187d","Type":"ContainerDied","Data":"a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f"} Oct 30 21:59:48.615399 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.615396 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n" event={"ID":"400bbd66-5849-48c0-8dcc-f152ccb7187d","Type":"ContainerDied","Data":"eb56c35ce71436e747ae5b7d380e215206e2b679f11241c6ff80a78e86bf8ce5"} Oct 30 21:59:48.615555 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.615420 5607 scope.go:122] "RemoveContainer" containerID="a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f" Oct 30 21:59:48.625644 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.625625 5607 scope.go:122] "RemoveContainer" containerID="a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f" Oct 30 21:59:48.625917 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:48.625899 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f\": container with ID starting with a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f not found: ID does not exist" containerID="a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f" Oct 30 21:59:48.626102 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.625923 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f"} err="failed to get container status \"a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f\": rpc error: code = NotFound desc = could not find container \"a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f\": container with ID starting with a0b330731cb4fa01a2ac0ac4b2652a546eee4a68a6696d2abf063c12d415db6f not found: ID does not exist" Oct 30 21:59:48.770259 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.770207 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n"] Oct 30 21:59:48.816745 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.816725 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4705e99c-7bfd5b8455-4z52n"] Oct 30 21:59:48.869357 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.869334 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377bb790-5dc9-4f58-8ec4-691f542a8c0f" path="/var/lib/kubelet/pods/377bb790-5dc9-4f58-8ec4-691f542a8c0f/volumes" Oct 30 21:59:48.869691 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:48.869677 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400bbd66-5849-48c0-8dcc-f152ccb7187d" path="/var/lib/kubelet/pods/400bbd66-5849-48c0-8dcc-f152ccb7187d/volumes" Oct 30 21:59:51.569668 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:51.569636 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 21:59:51.570253 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:51.569711 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 21:59:52.867522 ip-10-0-133-91 kubenswrapper[5607]: E1030 21:59:52.867484 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 21:59:54.624978 ip-10-0-133-91 kubenswrapper[5607]: I1030 21:59:54.619507 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 22:00:00.202990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.202963 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65"] Oct 30 22:00:00.203466 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.203452 5607 cpu_manager.go:414] "containerMap: removing container" podUID="400bbd66-5849-48c0-8dcc-f152ccb7187d" containerName="payload-pre-processing" Oct 30 22:00:00.203510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.203467 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400bbd66-5849-48c0-8dcc-f152ccb7187d" containerName="payload-pre-processing" Oct 30 22:00:00.203573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.203563 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="400bbd66-5849-48c0-8dcc-f152ccb7187d" containerName="payload-pre-processing" Oct 30 22:00:00.206184 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.206168 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl"] Oct 30 22:00:00.206298 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.206280 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.210096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.210074 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.213637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.212653 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9"] Oct 30 22:00:00.222814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.217051 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.267091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.267070 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:00:00.276481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.276457 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twtjf\" (UniqueName: \"kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.276598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.276497 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lng\" (UniqueName: \"kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.276651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.276628 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwzx\" (UniqueName: \"kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx\") pod \"maas-api-key-cleanup-29889960-72lpl\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.276694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.276668 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.276694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.276687 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.276859 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.276841 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-72lpl\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.312782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.312699 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65"] Oct 30 22:00:00.317044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.317015 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9"] Oct 30 22:00:00.317202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.317177 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl"] Oct 30 22:00:00.378239 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.378213 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-72lpl\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.378371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.378278 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twtjf\" (UniqueName: \"kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.378371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.378319 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lng\" (UniqueName: \"kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.378371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.378361 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwzx\" (UniqueName: \"kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx\") pod \"maas-api-key-cleanup-29889960-72lpl\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.378479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.378412 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.378479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.378439 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.379086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.379066 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-72lpl\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.385563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.379529 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.385563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.379546 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.437538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.437509 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwzx\" (UniqueName: \"kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx\") pod \"maas-api-key-cleanup-29889960-72lpl\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.437647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.437537 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lng\" (UniqueName: \"kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.437647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.437619 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twtjf\" (UniqueName: \"kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.522640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.522573 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:00.530293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.530275 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:00.536319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.536295 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:00.682092 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.682061 5607 generic.go:356] "Generic (PLEG): container finished" podID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerID="3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7" exitCode=0 Oct 30 22:00:00.682210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.682136 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" event={"ID":"60599210-aca4-4ef9-947a-3b21c31b55d9","Type":"ContainerDied","Data":"3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7"} Oct 30 22:00:00.682557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.682542 5607 scope.go:122] "RemoveContainer" containerID="3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7" Oct 30 22:00:00.683726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.683697 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:00.683838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.683779 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:00.683838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.683827 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:00.929215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.928580 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65"] Oct 30 22:00:00.933988 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:00.933942 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e4ce4b_db1d_4df4_ae1f_e1f9cd948e37.slice/crio-84da790f2d4a0e1a0525b9912fd9e002772f9c619b4f507b57fa0724cf1a1c15 WatchSource:0}: Error finding container 84da790f2d4a0e1a0525b9912fd9e002772f9c619b4f507b57fa0724cf1a1c15: Status 404 returned error can't find the container with id 84da790f2d4a0e1a0525b9912fd9e002772f9c619b4f507b57fa0724cf1a1c15 Oct 30 22:00:00.976697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.976668 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9"] Oct 30 22:00:00.977240 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:00.977129 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f94079_2cbb_4631_8734_b4eb544d316c.slice/crio-b6d70936f63d297c7d200a79654bee5a284a889b37ddee41d58df1cbecd2067b WatchSource:0}: Error finding container b6d70936f63d297c7d200a79654bee5a284a889b37ddee41d58df1cbecd2067b: Status 404 returned error can't find the container with id b6d70936f63d297c7d200a79654bee5a284a889b37ddee41d58df1cbecd2067b Oct 30 22:00:00.978640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:00.978617 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl"] Oct 30 22:00:00.979583 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:00.979560 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb7f2bd_a367_4a2b_881c_25255d8729c3.slice/crio-efe5835d6fe2dcccf405abdcd54ff17c9a9eebe0447455f6a7a0448e83e3b692 WatchSource:0}: Error finding container efe5835d6fe2dcccf405abdcd54ff17c9a9eebe0447455f6a7a0448e83e3b692: Status 404 returned error can't find the container with id efe5835d6fe2dcccf405abdcd54ff17c9a9eebe0447455f6a7a0448e83e3b692 Oct 30 22:00:01.689959 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.689928 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerStarted","Data":"0c9e3018004521ffc1e87f990db7a6a52b1b82b86fccf72f7de481483f03a758"} Oct 30 22:00:01.690434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.690002 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerStarted","Data":"b6d70936f63d297c7d200a79654bee5a284a889b37ddee41d58df1cbecd2067b"} Oct 30 22:00:01.691568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.691535 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerStarted","Data":"fb76c3a52c9ac5dc76fef64218ba8ce10004b8da99a5aad45a33ef7545ce20d1"} Oct 30 22:00:01.691695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.691588 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerStarted","Data":"84da790f2d4a0e1a0525b9912fd9e002772f9c619b4f507b57fa0724cf1a1c15"} Oct 30 22:00:01.693145 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.693116 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerStarted","Data":"f20a165c1a638ce3e59fecb1221d9d158b357a7ae65412a1f45347e44cef481c"} Oct 30 22:00:01.693277 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.693155 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerStarted","Data":"efe5835d6fe2dcccf405abdcd54ff17c9a9eebe0447455f6a7a0448e83e3b692"} Oct 30 22:00:01.722091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.712378 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" event={"ID":"60599210-aca4-4ef9-947a-3b21c31b55d9","Type":"ContainerStarted","Data":"4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6"} Oct 30 22:00:01.838993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.828459 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" podStartSLOduration=1.828435143 podStartE2EDuration="1.828435143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.804411367 +0000 UTC m=+781.204967490" watchObservedRunningTime="2026-10-30 22:00:01.828435143 +0000 UTC m=+781.228991266" Oct 30 22:00:01.897977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.897054 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 22:00:01.897977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.897190 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:00:01.897977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:01.897235 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:00:02.087997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.081018 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" podStartSLOduration=2.080996307 podStartE2EDuration="2.080996307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.07476239 +0000 UTC m=+781.475318512" watchObservedRunningTime="2026-10-30 22:00:02.080996307 +0000 UTC m=+781.481552429" Oct 30 22:00:02.191134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.191079 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" podStartSLOduration=2.191063152 podStartE2EDuration="2.191063152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.190169212 +0000 UTC m=+781.590725334" watchObservedRunningTime="2026-10-30 22:00:02.191063152 +0000 UTC m=+781.591619273" Oct 30 22:00:02.250008 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.230270 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548"] Oct 30 22:00:02.254300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.254271 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.294536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.294474 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548"] Oct 30 22:00:02.305271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.305246 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e524ce\"" Oct 30 22:00:02.351251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.351207 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q\") pod \"payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.351462 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.351365 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.452997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.452253 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.452997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.452388 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q\") pod \"payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.453778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.453756 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.568161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.563810 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q\") pod \"payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.568161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.567981 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:02.759981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.755626 5607 generic.go:356] "Generic (PLEG): container finished" podID="93f94079-2cbb-4631-8734-b4eb544d316c" containerID="0c9e3018004521ffc1e87f990db7a6a52b1b82b86fccf72f7de481483f03a758" exitCode=60 Oct 30 22:00:02.759981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.755729 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerDied","Data":"0c9e3018004521ffc1e87f990db7a6a52b1b82b86fccf72f7de481483f03a758"} Oct 30 22:00:02.759981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.756252 5607 scope.go:122] "RemoveContainer" containerID="0c9e3018004521ffc1e87f990db7a6a52b1b82b86fccf72f7de481483f03a758" Oct 30 22:00:02.769901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.769875 5607 generic.go:356] "Generic (PLEG): container finished" podID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerID="fb76c3a52c9ac5dc76fef64218ba8ce10004b8da99a5aad45a33ef7545ce20d1" exitCode=60 Oct 30 22:00:02.770042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.770023 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerDied","Data":"fb76c3a52c9ac5dc76fef64218ba8ce10004b8da99a5aad45a33ef7545ce20d1"} Oct 30 22:00:02.770323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.770305 5607 scope.go:122] "RemoveContainer" containerID="fb76c3a52c9ac5dc76fef64218ba8ce10004b8da99a5aad45a33ef7545ce20d1" Oct 30 22:00:02.772185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.772163 5607 generic.go:356] "Generic (PLEG): container finished" podID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerID="f20a165c1a638ce3e59fecb1221d9d158b357a7ae65412a1f45347e44cef481c" exitCode=60 Oct 30 22:00:02.772452 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.772435 5607 scope.go:122] "RemoveContainer" containerID="f20a165c1a638ce3e59fecb1221d9d158b357a7ae65412a1f45347e44cef481c" Oct 30 22:00:02.772552 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.772528 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerDied","Data":"f20a165c1a638ce3e59fecb1221d9d158b357a7ae65412a1f45347e44cef481c"} Oct 30 22:00:02.892512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.892481 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:00:02.892650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:02.892529 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:00:03.040246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.040175 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548"] Oct 30 22:00:03.045272 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:03.045242 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375990b8_fabe_4822_881d_d776174dd6ec.slice/crio-0476ab24e11b3c6cf892b3156556eecbd5f51ecb3ffd78a4b3671079ced118a1 WatchSource:0}: Error finding container 0476ab24e11b3c6cf892b3156556eecbd5f51ecb3ffd78a4b3671079ced118a1: Status 404 returned error can't find the container with id 0476ab24e11b3c6cf892b3156556eecbd5f51ecb3ffd78a4b3671079ced118a1 Oct 30 22:00:03.780309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.780269 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerStarted","Data":"6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc"} Oct 30 22:00:03.782719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.782689 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerStarted","Data":"558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749"} Oct 30 22:00:03.784490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.784464 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerStarted","Data":"c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0"} Oct 30 22:00:03.785736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.785715 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" event={"ID":"375990b8-fabe-4822-881d-d776174dd6ec","Type":"ContainerStarted","Data":"c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5"} Oct 30 22:00:03.785821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.785758 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" event={"ID":"375990b8-fabe-4822-881d-d776174dd6ec","Type":"ContainerStarted","Data":"0476ab24e11b3c6cf892b3156556eecbd5f51ecb3ffd78a4b3671079ced118a1"} Oct 30 22:00:03.785889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.785868 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:03.892199 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.892173 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:00:03.892328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:03.892221 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:00:04.149934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.149810 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" podStartSLOduration=2.149792966 podStartE2EDuration="2.149792966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.036237198 +0000 UTC m=+783.436793332" watchObservedRunningTime="2026-10-30 22:00:04.149792966 +0000 UTC m=+783.550349090" Oct 30 22:00:04.792644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.792609 5607 generic.go:356] "Generic (PLEG): container finished" podID="93f94079-2cbb-4631-8734-b4eb544d316c" containerID="6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc" exitCode=60 Oct 30 22:00:04.793162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.792706 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerDied","Data":"6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc"} Oct 30 22:00:04.793162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.792772 5607 scope.go:122] "RemoveContainer" containerID="0c9e3018004521ffc1e87f990db7a6a52b1b82b86fccf72f7de481483f03a758" Oct 30 22:00:04.793162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.792931 5607 scope.go:122] "RemoveContainer" containerID="6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc" Oct 30 22:00:04.793349 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:04.793224 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9_odh-ai-gateway-infra(93f94079-2cbb-4631-8734-b4eb544d316c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" Oct 30 22:00:04.823976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.820409 5607 generic.go:356] "Generic (PLEG): container finished" podID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerID="558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749" exitCode=60 Oct 30 22:00:04.823976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.820524 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerDied","Data":"558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749"} Oct 30 22:00:04.823976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.820903 5607 scope.go:122] "RemoveContainer" containerID="558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749" Oct 30 22:00:04.823976 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:04.821203 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65_odh-ai-gateway-infra(57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" Oct 30 22:00:04.849979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.847312 5607 generic.go:356] "Generic (PLEG): container finished" podID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerID="c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0" exitCode=60 Oct 30 22:00:04.849979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.848339 5607 scope.go:122] "RemoveContainer" containerID="c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0" Oct 30 22:00:04.849979 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:04.848579 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-72lpl_odh-ai-gateway-infra(7cb7f2bd-a367-4a2b-881c-25255d8729c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" Oct 30 22:00:04.849979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.848626 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerDied","Data":"c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0"} Oct 30 22:00:04.850973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.850476 5607 scope.go:122] "RemoveContainer" containerID="fb76c3a52c9ac5dc76fef64218ba8ce10004b8da99a5aad45a33ef7545ce20d1" Oct 30 22:00:04.862943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.862928 5607 scope.go:122] "RemoveContainer" containerID="f20a165c1a638ce3e59fecb1221d9d158b357a7ae65412a1f45347e44cef481c" Oct 30 22:00:04.891820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.891798 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:00:04.891914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:04.891840 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:00:05.862645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:05.862155 5607 scope.go:122] "RemoveContainer" containerID="6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc" Oct 30 22:00:05.862645 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:05.862525 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9_odh-ai-gateway-infra(93f94079-2cbb-4631-8734-b4eb544d316c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" Oct 30 22:00:05.865737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:05.865712 5607 scope.go:122] "RemoveContainer" containerID="558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749" Oct 30 22:00:05.866232 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:05.866207 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65_odh-ai-gateway-infra(57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" Oct 30 22:00:05.868010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:05.867992 5607 scope.go:122] "RemoveContainer" containerID="c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0" Oct 30 22:00:05.868207 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:05.868187 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-72lpl_odh-ai-gateway-infra(7cb7f2bd-a367-4a2b-881c-25255d8729c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" Oct 30 22:00:05.891880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:05.891863 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:00:05.892040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:05.892022 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:00:07.010979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:07.010266 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 22:00:07.019977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:07.011617 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 22:00:07.022976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:07.020158 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 22:00:10.896912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:10.896827 5607 generic.go:356] "Generic (PLEG): container finished" podID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerID="e0dc21d5b3a684ef53948b055fadc035aba7a8b727687531262a4b826fbb82b4" exitCode=0 Oct 30 22:00:10.897433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:10.896908 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" event={"ID":"1511ebe9-bdff-44e5-a614-9e70273e7e33","Type":"ContainerDied","Data":"e0dc21d5b3a684ef53948b055fadc035aba7a8b727687531262a4b826fbb82b4"} Oct 30 22:00:10.897433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:10.897321 5607 scope.go:122] "RemoveContainer" containerID="e0dc21d5b3a684ef53948b055fadc035aba7a8b727687531262a4b826fbb82b4" Oct 30 22:00:10.898505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:10.898479 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:10.898565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:10.898544 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:10.898601 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:10.898576 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:11.922977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:11.922271 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" event={"ID":"1511ebe9-bdff-44e5-a614-9e70273e7e33","Type":"ContainerStarted","Data":"c9904186f9167e07d65605b921c1b43025acdea9059b8f1f8b3da2b63396b1e7"} Oct 30 22:00:12.921560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:12.921530 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 22:00:12.921936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:12.921913 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:00:12.922163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:12.922139 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:00:13.912323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:13.912288 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:00:13.912903 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:13.912873 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:00:14.851239 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:14.851207 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:00:14.863398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:14.863373 5607 scope.go:122] "RemoveContainer" containerID="558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749" Oct 30 22:00:14.938974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:14.925392 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:00:14.938974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:14.925449 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:00:15.863254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:15.863224 5607 scope.go:122] "RemoveContainer" containerID="c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0" Oct 30 22:00:15.923973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:15.917658 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:00:15.923973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:15.917710 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:00:15.970291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:15.970251 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerStarted","Data":"d1db44b843ce6c9b64280e949151a0e9080b246fd4a559b9b4960a7c5a665861"} Oct 30 22:00:16.912279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.912244 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:00:16.912594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.912563 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:00:16.978356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.978326 5607 generic.go:356] "Generic (PLEG): container finished" podID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerID="d1db44b843ce6c9b64280e949151a0e9080b246fd4a559b9b4960a7c5a665861" exitCode=60 Oct 30 22:00:16.978773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.978415 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerDied","Data":"d1db44b843ce6c9b64280e949151a0e9080b246fd4a559b9b4960a7c5a665861"} Oct 30 22:00:16.978773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.978502 5607 scope.go:122] "RemoveContainer" containerID="558543a16f8c2c171626192d573e053b105a4e66c0deb041f72d3990f7c10749" Oct 30 22:00:16.978935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.978910 5607 scope.go:122] "RemoveContainer" containerID="d1db44b843ce6c9b64280e949151a0e9080b246fd4a559b9b4960a7c5a665861" Oct 30 22:00:16.979239 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:16.979208 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65_odh-ai-gateway-infra(57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" Oct 30 22:00:16.980590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.980564 5607 generic.go:356] "Generic (PLEG): container finished" podID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerID="ba99b4ce9abb5ceb36aa54aa72f10734b1f80fca4e4b6fa1f447bd195aeb4c0b" exitCode=60 Oct 30 22:00:16.980703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.980622 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerDied","Data":"ba99b4ce9abb5ceb36aa54aa72f10734b1f80fca4e4b6fa1f447bd195aeb4c0b"} Oct 30 22:00:16.980974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:16.980937 5607 scope.go:122] "RemoveContainer" containerID="ba99b4ce9abb5ceb36aa54aa72f10734b1f80fca4e4b6fa1f447bd195aeb4c0b" Oct 30 22:00:16.981184 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:16.981165 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-72lpl_odh-ai-gateway-infra(7cb7f2bd-a367-4a2b-881c-25255d8729c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" Oct 30 22:00:17.026231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:17.025174 5607 scope.go:122] "RemoveContainer" containerID="c9d216126b2afaa34ee81c5db6bb66c52e716a5e1b49ec726801c95f42ee9bd0" Oct 30 22:00:17.124373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:17.124333 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65"] Oct 30 22:00:17.497838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:17.497810 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 22:00:17.863419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:17.863384 5607 scope.go:122] "RemoveContainer" containerID="6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc" Oct 30 22:00:17.922520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:17.922489 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 22:00:17.992336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:17.992312 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 22:00:18.031266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.031232 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb"] Oct 30 22:00:18.037048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.037025 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.087774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.087700 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e65019-dockercfg-r4wf6\"" Oct 30 22:00:18.089074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.089047 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e65019\"" Oct 30 22:00:18.090424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.090189 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.090424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.090242 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2ng\" (UniqueName: \"kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng\") pod \"payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.091351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.091327 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb"] Oct 30 22:00:18.160223 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.160192 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:18.177483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.177456 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8"] Oct 30 22:00:18.178074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178057 5607 cpu_manager.go:414] "containerMap: removing container" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178076 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178098 5607 cpu_manager.go:414] "containerMap: removing container" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178106 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178131 5607 cpu_manager.go:414] "containerMap: removing container" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178138 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178292 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.178424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.178314 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:18.182946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.182925 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.190843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.190822 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca\") pod \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " Oct 30 22:00:18.191011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.190991 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng\" (UniqueName: \"kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng\") pod \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\" (UID: \"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37\") " Oct 30 22:00:18.191197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.191176 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca" pod "57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" (UID: "57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.191274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.191256 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.191316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.191304 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2ng\" (UniqueName: \"kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng\") pod \"payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.191356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.191345 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.191457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.191440 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lccr\" (UniqueName: \"kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr\") pod \"payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.191512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.191487 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.192240 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.192213 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.193423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.193397 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng" pod "57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" (UID: "57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37"). InnerVolumeSpecName "kube-api-access-z8lng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.208309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.208287 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl"] Oct 30 22:00:18.236556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.236378 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8"] Oct 30 22:00:18.250157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.250129 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2ng\" (UniqueName: \"kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng\") pod \"payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.292335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.292292 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.292503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.292395 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lccr\" (UniqueName: \"kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr\") pod \"payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.292503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.292485 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lng\" (UniqueName: \"kubernetes.io/projected/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37-kube-api-access-z8lng\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.293108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.293089 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.351477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.351451 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lccr\" (UniqueName: \"kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr\") pod \"payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.353864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.353843 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:18.359355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.359335 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:18.394071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.393943 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca\") pod \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " Oct 30 22:00:18.394233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.394155 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx\" (UniqueName: \"kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx\") pod \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\" (UID: \"7cb7f2bd-a367-4a2b-881c-25255d8729c3\") " Oct 30 22:00:18.394710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.394645 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca" pod "7cb7f2bd-a367-4a2b-881c-25255d8729c3" (UID: "7cb7f2bd-a367-4a2b-881c-25255d8729c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.400969 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.400908 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx" pod "7cb7f2bd-a367-4a2b-881c-25255d8729c3" (UID: "7cb7f2bd-a367-4a2b-881c-25255d8729c3"). InnerVolumeSpecName "kube-api-access-tgwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.497621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.497538 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:18.498352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.498330 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgwzx\" (UniqueName: \"kubernetes.io/projected/7cb7f2bd-a367-4a2b-881c-25255d8729c3-kube-api-access-tgwzx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.498489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.498355 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7f2bd-a367-4a2b-881c-25255d8729c3-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.744030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.744004 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb"] Oct 30 22:00:18.746060 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:18.746022 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fe4eec_a061_47ae_8ac5_dc9b7c4cb94f.slice/crio-fbcd58a94d2e5df63eee50bb848024e68c81fd5044a9445ad4e28091954a2019 WatchSource:0}: Error finding container fbcd58a94d2e5df63eee50bb848024e68c81fd5044a9445ad4e28091954a2019: Status 404 returned error can't find the container with id fbcd58a94d2e5df63eee50bb848024e68c81fd5044a9445ad4e28091954a2019 Oct 30 22:00:18.900582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:18.900558 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8"] Oct 30 22:00:19.015268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.015235 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" event={"ID":"36ca4a4f-814f-4467-850f-7c72ca21d55c","Type":"ContainerStarted","Data":"d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b"} Oct 30 22:00:19.016294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.015766 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" event={"ID":"36ca4a4f-814f-4467-850f-7c72ca21d55c","Type":"ContainerStarted","Data":"cc8f51d5e27d4d5a438d2da26247d3678a4156ed71cba35c249f02b419400b05"} Oct 30 22:00:19.016294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.015811 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:19.020646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.020575 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerStarted","Data":"743a99cdebd4211db0b70c44542c5582b843334451c217567585ad9a41d2848c"} Oct 30 22:00:19.022280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.022252 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" event={"ID":"57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37","Type":"ContainerDied","Data":"84da790f2d4a0e1a0525b9912fd9e002772f9c619b4f507b57fa0724cf1a1c15"} Oct 30 22:00:19.022376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.022306 5607 scope.go:122] "RemoveContainer" containerID="d1db44b843ce6c9b64280e949151a0e9080b246fd4a559b9b4960a7c5a665861" Oct 30 22:00:19.022427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.022413 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65" Oct 30 22:00:19.025393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.025364 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" Oct 30 22:00:19.025505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.025411 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl" event={"ID":"7cb7f2bd-a367-4a2b-881c-25255d8729c3","Type":"ContainerDied","Data":"efe5835d6fe2dcccf405abdcd54ff17c9a9eebe0447455f6a7a0448e83e3b692"} Oct 30 22:00:19.027446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.027424 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" event={"ID":"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f","Type":"ContainerStarted","Data":"39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7"} Oct 30 22:00:19.027558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.027463 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" event={"ID":"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f","Type":"ContainerStarted","Data":"fbcd58a94d2e5df63eee50bb848024e68c81fd5044a9445ad4e28091954a2019"} Oct 30 22:00:19.027558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.027528 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:19.060121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.060099 5607 scope.go:122] "RemoveContainer" containerID="ba99b4ce9abb5ceb36aa54aa72f10734b1f80fca4e4b6fa1f447bd195aeb4c0b" Oct 30 22:00:19.143046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.143001 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" podStartSLOduration=1.142988068 podStartE2EDuration="1.142988068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:19.14118876 +0000 UTC m=+798.541744881" watchObservedRunningTime="2026-10-30 22:00:19.142988068 +0000 UTC m=+798.543544210" Oct 30 22:00:19.382220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.382143 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65"] Oct 30 22:00:19.394674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.394649 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889960-5wj65"] Oct 30 22:00:19.498000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.497972 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl"] Oct 30 22:00:19.514034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.514013 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-72lpl"] Oct 30 22:00:19.625914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:19.625858 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" podStartSLOduration=2.625840174 podStartE2EDuration="2.625840174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:19.623031795 +0000 UTC m=+799.023587917" watchObservedRunningTime="2026-10-30 22:00:19.625840174 +0000 UTC m=+799.026396301" Oct 30 22:00:20.037991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.037930 5607 generic.go:356] "Generic (PLEG): container finished" podID="93f94079-2cbb-4631-8734-b4eb544d316c" containerID="743a99cdebd4211db0b70c44542c5582b843334451c217567585ad9a41d2848c" exitCode=60 Oct 30 22:00:20.038373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.038049 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerDied","Data":"743a99cdebd4211db0b70c44542c5582b843334451c217567585ad9a41d2848c"} Oct 30 22:00:20.038373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.038144 5607 scope.go:122] "RemoveContainer" containerID="6360749e7a461fa2ac2dedc729210a963045e587903bfe69480fea5c767189fc" Oct 30 22:00:20.038722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.038580 5607 scope.go:122] "RemoveContainer" containerID="743a99cdebd4211db0b70c44542c5582b843334451c217567585ad9a41d2848c" Oct 30 22:00:20.038904 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:20.038878 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9_odh-ai-gateway-infra(93f94079-2cbb-4631-8734-b4eb544d316c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" Oct 30 22:00:20.268055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.268023 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9"] Oct 30 22:00:20.869206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.869178 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" path="/var/lib/kubelet/pods/57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37/volumes" Oct 30 22:00:20.869664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:20.869646 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" path="/var/lib/kubelet/pods/7cb7f2bd-a367-4a2b-881c-25255d8729c3/volumes" Oct 30 22:00:21.054598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.054574 5607 generic.go:356] "Generic (PLEG): container finished" podID="2e271101-b9ba-4b33-8862-88373435f991" containerID="3fd5e1fc3c708d32f91e95624f4d27977d20e57650809759074725a1aaab4b1e" exitCode=0 Oct 30 22:00:21.054914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.054609 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" event={"ID":"2e271101-b9ba-4b33-8862-88373435f991","Type":"ContainerDied","Data":"3fd5e1fc3c708d32f91e95624f4d27977d20e57650809759074725a1aaab4b1e"} Oct 30 22:00:21.054978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.054938 5607 scope.go:122] "RemoveContainer" containerID="3fd5e1fc3c708d32f91e95624f4d27977d20e57650809759074725a1aaab4b1e" Oct 30 22:00:21.056207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.056164 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:21.056305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.056252 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:21.056305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.056293 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:21.203385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.203363 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:21.223482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.223456 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca\") pod \"93f94079-2cbb-4631-8734-b4eb544d316c\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " Oct 30 22:00:21.223638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.223522 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf\" (UniqueName: \"kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf\") pod \"93f94079-2cbb-4631-8734-b4eb544d316c\" (UID: \"93f94079-2cbb-4631-8734-b4eb544d316c\") " Oct 30 22:00:21.223920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.223897 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca" pod "93f94079-2cbb-4631-8734-b4eb544d316c" (UID: "93f94079-2cbb-4631-8734-b4eb544d316c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.224034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.224014 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f94079-2cbb-4631-8734-b4eb544d316c-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.226313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.226284 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf" pod "93f94079-2cbb-4631-8734-b4eb544d316c" (UID: "93f94079-2cbb-4631-8734-b4eb544d316c"). InnerVolumeSpecName "kube-api-access-twtjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.324592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:21.324564 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twtjf\" (UniqueName: \"kubernetes.io/projected/93f94079-2cbb-4631-8734-b4eb544d316c-kube-api-access-twtjf\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:22.063431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.063400 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" event={"ID":"2e271101-b9ba-4b33-8862-88373435f991","Type":"ContainerStarted","Data":"70b936cc36991bd4e93f61ff5753d9b14a0cb7a861a0058a535f43641c517021"} Oct 30 22:00:22.064771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.064743 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" Oct 30 22:00:22.064771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.064761 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9" event={"ID":"93f94079-2cbb-4631-8734-b4eb544d316c","Type":"ContainerDied","Data":"b6d70936f63d297c7d200a79654bee5a284a889b37ddee41d58df1cbecd2067b"} Oct 30 22:00:22.064986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.064809 5607 scope.go:122] "RemoveContainer" containerID="743a99cdebd4211db0b70c44542c5582b843334451c217567585ad9a41d2848c" Oct 30 22:00:22.227300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.227261 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 22:00:22.227490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.227466 5607 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:00:22.227543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.227511 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:00:22.287190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.287166 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9"] Oct 30 22:00:22.288899 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.288880 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889960-j6wl9"] Oct 30 22:00:22.863318 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:22.863279 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:00:22.869688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:22.869660 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" path="/var/lib/kubelet/pods/93f94079-2cbb-4631-8734-b4eb544d316c/volumes" Oct 30 22:00:23.227468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:23.227429 5607 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:00:23.227892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:23.227475 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:00:24.227475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:24.227436 5607 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:00:24.227865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:24.227497 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:00:25.227504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:25.227468 5607 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.227908 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:25.227521 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:00:26.227413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:26.227384 5607 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.227582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:26.227433 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:00:27.231719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:27.231687 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 22:00:27.232121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:27.232105 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 22:00:27.233049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:27.233030 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 22:00:30.044882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:30.044849 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:00:30.045356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:30.044927 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:00:36.467426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467393 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6"] Oct 30 22:00:36.467867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467854 5607 cpu_manager.go:414] "containerMap: removing container" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.467867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467868 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467880 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467885 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467904 5607 cpu_manager.go:414] "containerMap: removing container" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467910 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467927 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467931 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467937 5607 cpu_manager.go:414] "containerMap: removing container" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.467945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.467942 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.468215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.468031 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.468215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.468038 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.468215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.468047 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.468215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.468053 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e4ce4b-db1d-4df4-ae1f-e1f9cd948e37" containerName="cleanup" Oct 30 22:00:36.468215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.468060 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f94079-2cbb-4631-8734-b4eb544d316c" containerName="cleanup" Oct 30 22:00:36.468215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.468066 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.472996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.472980 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.514776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.514750 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58"] Oct 30 22:00:36.515404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.515378 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.515404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.515397 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.515577 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.515504 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb7f2bd-a367-4a2b-881c-25255d8729c3" containerName="cleanup" Oct 30 22:00:36.518721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.518703 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.529502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.529480 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e524ce-openshift-default-dockercfg-zs996\"" Oct 30 22:00:36.569105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.569041 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6"] Oct 30 22:00:36.571818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.571791 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.571932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.571833 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.571932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.571854 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572029 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572090 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572139 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572124 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572200 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572186 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572235 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572338 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572386 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpw4\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572452 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572480 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rkx\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572499 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572518 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572622 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572671 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.572730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572699 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.572770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.572728 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.618247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.618225 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58"] Oct 30 22:00:36.673692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673669 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.673803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673710 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.673803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673732 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpw4\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.673803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673762 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.673803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673786 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rkx\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.673803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673803 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673819 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673851 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.673929 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674215 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674248 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674265 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674135 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674264 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674307 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674314 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674347 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674371 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674388 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674424 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674472 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674481 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674503 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.674627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674537 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.675056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674829 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.675056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.674879 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.675465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.675439 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.675578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.675494 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.677005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.676980 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.677174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.677158 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.677216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.677175 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.677522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.677504 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.732165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.732099 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.732273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.732254 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.732317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.732270 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rkx\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.732317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.732281 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpw4\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:36.807940 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.807916 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:36.835035 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:36.835010 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:37.230194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.230165 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6"] Oct 30 22:00:37.231793 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:37.231766 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1afb645_4dd0_4d16_8899_e65634921b01.slice/crio-9b9113e2dae7fba49e5b5e6700dbaa14f593cadba07e2ca02a28aff0418d5213 WatchSource:0}: Error finding container 9b9113e2dae7fba49e5b5e6700dbaa14f593cadba07e2ca02a28aff0418d5213: Status 404 returned error can't find the container with id 9b9113e2dae7fba49e5b5e6700dbaa14f593cadba07e2ca02a28aff0418d5213 Oct 30 22:00:37.234105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.234073 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:37.234164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.234135 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:37.234204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.234163 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:37.284162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.284142 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58"] Oct 30 22:00:37.286474 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:00:37.286453 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421866cc_91cc_411e_a1bc_d6ea9f7e1f2a.slice/crio-5949323faec14a99ae45500787babd7988d897b1e200464459c9acaaea7b7a4a WatchSource:0}: Error finding container 5949323faec14a99ae45500787babd7988d897b1e200464459c9acaaea7b7a4a: Status 404 returned error can't find the container with id 5949323faec14a99ae45500787babd7988d897b1e200464459c9acaaea7b7a4a Oct 30 22:00:37.288933 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.288903 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:37.289054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.288996 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:37.289054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:37.289039 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:38.160325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.160279 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" event={"ID":"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a","Type":"ContainerStarted","Data":"d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87"} Oct 30 22:00:38.160834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.160374 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" event={"ID":"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a","Type":"ContainerStarted","Data":"5949323faec14a99ae45500787babd7988d897b1e200464459c9acaaea7b7a4a"} Oct 30 22:00:38.161975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.161925 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" event={"ID":"f1afb645-4dd0-4d16-8899-e65634921b01","Type":"ContainerStarted","Data":"fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34"} Oct 30 22:00:38.162119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.161997 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" event={"ID":"f1afb645-4dd0-4d16-8899-e65634921b01","Type":"ContainerStarted","Data":"9b9113e2dae7fba49e5b5e6700dbaa14f593cadba07e2ca02a28aff0418d5213"} Oct 30 22:00:38.347235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.347185 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podStartSLOduration=2.347171948 podStartE2EDuration="2.347171948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:38.345367017 +0000 UTC m=+817.745923139" watchObservedRunningTime="2026-10-30 22:00:38.347171948 +0000 UTC m=+817.747728069" Oct 30 22:00:38.463622 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.463574 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podStartSLOduration=2.463559983 podStartE2EDuration="2.463559983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:38.461263559 +0000 UTC m=+817.861819719" watchObservedRunningTime="2026-10-30 22:00:38.463559983 +0000 UTC m=+817.864116104" Oct 30 22:00:38.809033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.808716 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:38.818978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.818086 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:00:38.818978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.818171 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:00:38.845639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.845329 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:38.847528 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.847497 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:00:38.847633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:38.847553 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:00:39.814979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:39.808365 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:00:39.814979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:39.808425 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:00:39.845014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:39.843349 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:00:39.845014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:39.843405 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:00:40.821978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:40.810482 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:00:40.821978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:40.810553 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:00:40.839983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:40.839412 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:00:40.839983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:40.839463 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:00:41.816978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:41.812322 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:00:41.816978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:41.812381 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:00:41.838339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:41.838305 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:00:41.838937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:41.838906 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:00:42.810359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:42.810320 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:00:42.810651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:42.810622 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:00:42.838341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:42.838310 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:00:42.838596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:42.838566 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:00:43.812807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:43.812771 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:43.813085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:43.813065 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:43.813826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:43.813809 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:00:43.839770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:43.839746 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:43.840091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:43.839982 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:43.840574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:00:43.840559 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:00:52.863258 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:00:52.863217 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:01:17.212145 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.212106 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h"] Oct 30 22:01:17.215936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.215907 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.258422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.258396 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8zd\" (UniqueName: \"kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.258549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.258431 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.258549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.258456 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.258549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.258534 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.258652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.258613 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.258652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.258642 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.310235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.310207 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"kube-root-ca.crt\"" Oct 30 22:01:17.315373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.315347 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"simulator-w4-e524ce-kserve-self-signed-certs\"" Oct 30 22:01:17.315373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.315352 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"default-dockercfg-ljjzd\"" Oct 30 22:01:17.315373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.315354 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"openshift-service-ca.crt\"" Oct 30 22:01:17.319545 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.319524 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h"] Oct 30 22:01:17.360372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.360344 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8zd\" (UniqueName: \"kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.361741 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.361804 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.361831 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.361893 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.361939 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.362441 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.362458 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.363268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.362679 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.376491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.376109 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.379615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.379525 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.422867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.422833 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8zd\" (UniqueName: \"kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd\") pod \"simulator-w4-e524ce-kserve-5b6cc56dc-65s7h\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.527824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.527761 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:17.898359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:17.898337 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h"] Oct 30 22:01:17.900230 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:01:17.900203 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod450a2dde_cc87_40f6_bae8_ce0d32dd1428.slice/crio-e16c80b8043733b24cf5156535066e4229f2277f3a8fcf92a875055375fecdd0 WatchSource:0}: Error finding container e16c80b8043733b24cf5156535066e4229f2277f3a8fcf92a875055375fecdd0: Status 404 returned error can't find the container with id e16c80b8043733b24cf5156535066e4229f2277f3a8fcf92a875055375fecdd0 Oct 30 22:01:18.389696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:18.389661 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" event={"ID":"450a2dde-cc87-40f6-bae8-ce0d32dd1428","Type":"ContainerStarted","Data":"8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198"} Oct 30 22:01:18.390098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:18.389708 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" event={"ID":"450a2dde-cc87-40f6-bae8-ce0d32dd1428","Type":"ContainerStarted","Data":"e16c80b8043733b24cf5156535066e4229f2277f3a8fcf92a875055375fecdd0"} Oct 30 22:01:18.505143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:18.505086 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podStartSLOduration=1.505069943 podStartE2EDuration="1.505069943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:18.50475061 +0000 UTC m=+857.905306732" watchObservedRunningTime="2026-10-30 22:01:18.505069943 +0000 UTC m=+857.905626066" Oct 30 22:01:18.527944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:18.527901 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:22.864067 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:01:22.864032 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:01:27.528046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:27.528015 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:27.542982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:27.542459 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:28.459562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:28.459537 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:01:52.862635 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:01:52.862596 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:01:56.626907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.626876 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6"] Oct 30 22:01:56.630768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.630751 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.660222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.660199 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.660359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.660245 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.660425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.660378 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.660464 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.660451 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.660498 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.660474 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4j6\" (UniqueName: \"kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.660547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.660510 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.727218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.727187 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"default-dockercfg-wq5ll\"" Oct 30 22:01:56.727382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.727332 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"premium-w0-d09fff-kserve-self-signed-certs\"" Oct 30 22:01:56.727707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.727689 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"openshift-service-ca.crt\"" Oct 30 22:01:56.727782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.727758 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"kube-root-ca.crt\"" Oct 30 22:01:56.729381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.729358 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6"] Oct 30 22:01:56.761356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761329 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761386 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761418 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4j6\" (UniqueName: \"kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761467 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761631 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761590 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761641 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761734 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761777 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.761930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.761912 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.763685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.763665 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.764200 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.764181 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.820260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.820232 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4j6\" (UniqueName: \"kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6\") pod \"premium-w0-d09fff-kserve-57455dc7f-jgbg6\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:56.943346 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:56.943318 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:01:57.311250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:57.311211 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6"] Oct 30 22:01:57.313432 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:01:57.313387 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfaba7da_6325_41c6_87da_db337fffd973.slice/crio-fc247f9c7a445e78b5cedc6f773b6933e3adc5b269dff4ae312f80a4de124670 WatchSource:0}: Error finding container fc247f9c7a445e78b5cedc6f773b6933e3adc5b269dff4ae312f80a4de124670: Status 404 returned error can't find the container with id fc247f9c7a445e78b5cedc6f773b6933e3adc5b269dff4ae312f80a4de124670 Oct 30 22:01:57.610413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:57.610326 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" event={"ID":"dfaba7da-6325-41c6-87da-db337fffd973","Type":"ContainerStarted","Data":"85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171"} Oct 30 22:01:57.610413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:57.610387 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" event={"ID":"dfaba7da-6325-41c6-87da-db337fffd973","Type":"ContainerStarted","Data":"fc247f9c7a445e78b5cedc6f773b6933e3adc5b269dff4ae312f80a4de124670"} Oct 30 22:01:57.722741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:57.722690 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podStartSLOduration=1.722676834 podStartE2EDuration="1.722676834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:57.720648466 +0000 UTC m=+897.121204584" watchObservedRunningTime="2026-10-30 22:01:57.722676834 +0000 UTC m=+897.123233339" Oct 30 22:01:57.943887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:01:57.943860 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:02:00.965442 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:00.965417 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:02:00.977347 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:00.977328 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:02:06.943768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:06.943734 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:02:06.956541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:06.956520 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:02:07.677297 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:07.677270 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:02:22.862179 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:02:22.862143 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:02:26.951711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:26.951682 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6"] Oct 30 22:02:26.955884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:26.955862 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.010097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.010074 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6"] Oct 30 22:02:27.054250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.054220 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:27.054401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.054256 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:02:27.083151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.083127 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2tt\" (UniqueName: \"kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.083238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.083171 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.083238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.083231 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.184388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.184362 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.184489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.184471 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2tt\" (UniqueName: \"kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.184544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.184530 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.185192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.185176 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.186880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.186863 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.242632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.242580 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2tt\" (UniqueName: \"kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt\") pod \"maas-api-revoke-sub-d8d0d532-pcts6\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.268519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.268495 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:27.455764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.455734 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz"] Oct 30 22:02:27.455975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.455933 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" containerID="cri-o://457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df" gracePeriod=30 Oct 30 22:02:27.642793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.642766 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6"] Oct 30 22:02:27.644592 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:02:27.644564 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9a25bf_6af5_4690_b0ed_48d5522566b2.slice/crio-c432335a720bb13ba53a7b7b4381c39b90b687121253ac5ae4d8265e542f1027 WatchSource:0}: Error finding container c432335a720bb13ba53a7b7b4381c39b90b687121253ac5ae4d8265e542f1027: Status 404 returned error can't find the container with id c432335a720bb13ba53a7b7b4381c39b90b687121253ac5ae4d8265e542f1027 Oct 30 22:02:27.772547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:27.772521 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" event={"ID":"0e9a25bf-6af5-4690-b0ed-48d5522566b2","Type":"ContainerStarted","Data":"c432335a720bb13ba53a7b7b4381c39b90b687121253ac5ae4d8265e542f1027"} Oct 30 22:02:28.780357 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:28.780289 5607 generic.go:356] "Generic (PLEG): container finished" podID="0e9a25bf-6af5-4690-b0ed-48d5522566b2" containerID="9aca96536fe814afb96d3bf37231c026448549eaf0146714ce131b47c86bc1af" exitCode=0 Oct 30 22:02:28.780706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:28.780380 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" event={"ID":"0e9a25bf-6af5-4690-b0ed-48d5522566b2","Type":"ContainerDied","Data":"9aca96536fe814afb96d3bf37231c026448549eaf0146714ce131b47c86bc1af"} Oct 30 22:02:29.937786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:29.937764 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:30.012737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.012713 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca\") pod \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " Oct 30 22:02:30.012844 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.012759 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca\") pod \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " Oct 30 22:02:30.012885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.012877 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt\" (UniqueName: \"kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt\") pod \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\" (UID: \"0e9a25bf-6af5-4690-b0ed-48d5522566b2\") " Oct 30 22:02:30.013173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.013149 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca" pod "0e9a25bf-6af5-4690-b0ed-48d5522566b2" (UID: "0e9a25bf-6af5-4690-b0ed-48d5522566b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:30.014879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.014858 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca" pod "0e9a25bf-6af5-4690-b0ed-48d5522566b2" (UID: "0e9a25bf-6af5-4690-b0ed-48d5522566b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:30.015067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.015044 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt" pod "0e9a25bf-6af5-4690-b0ed-48d5522566b2" (UID: "0e9a25bf-6af5-4690-b0ed-48d5522566b2"). InnerVolumeSpecName "kube-api-access-pz2tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:30.114786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.114721 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2tt\" (UniqueName: \"kubernetes.io/projected/0e9a25bf-6af5-4690-b0ed-48d5522566b2-kube-api-access-pz2tt\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:30.114786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.114747 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:30.114786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.114758 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a25bf-6af5-4690-b0ed-48d5522566b2-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:30.792796 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.792771 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" Oct 30 22:02:30.792981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.792810 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6" event={"ID":"0e9a25bf-6af5-4690-b0ed-48d5522566b2","Type":"ContainerDied","Data":"c432335a720bb13ba53a7b7b4381c39b90b687121253ac5ae4d8265e542f1027"} Oct 30 22:02:30.792981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:30.792887 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c432335a720bb13ba53a7b7b4381c39b90b687121253ac5ae4d8265e542f1027" Oct 30 22:02:32.718753 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.718696 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 22:02:32.823909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.823878 5607 generic.go:356] "Generic (PLEG): container finished" podID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerID="457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df" exitCode=0 Oct 30 22:02:32.824073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.823921 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" Oct 30 22:02:32.824073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.823987 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" event={"ID":"110e6a5c-b832-44b6-bf4d-fb91d24fb59a","Type":"ContainerDied","Data":"457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df"} Oct 30 22:02:32.824073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.824041 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz" event={"ID":"110e6a5c-b832-44b6-bf4d-fb91d24fb59a","Type":"ContainerDied","Data":"4d3ff5d82b3e38e00adcab88eb0353207508d17e5c536d8b459efd04bea1bcd6"} Oct 30 22:02:32.824073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.824068 5607 scope.go:122] "RemoveContainer" containerID="457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df" Oct 30 22:02:32.833260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.833245 5607 scope.go:122] "RemoveContainer" containerID="457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df" Oct 30 22:02:32.833542 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:02:32.833524 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df\": container with ID starting with 457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df not found: ID does not exist" containerID="457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df" Oct 30 22:02:32.833598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.833549 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df"} err="failed to get container status \"457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df\": rpc error: code = NotFound desc = could not find container \"457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df\": container with ID starting with 457fc44939815579bbd65e5b526103f79bf20be5cd8e17b64f875460c809e5df not found: ID does not exist" Oct 30 22:02:32.869263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869241 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.869369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869294 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.869369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869322 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.869494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869372 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.869494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869428 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.869604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869537 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.869840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869725 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.869840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869792 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.869840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869825 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.870238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869940 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.870238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.869982 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.870238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870038 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket\") pod \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\" (UID: \"110e6a5c-b832-44b6-bf4d-fb91d24fb59a\") " Oct 30 22:02:32.870400 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870350 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.870618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870596 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:32.870722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870663 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.870722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870685 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.870722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870698 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.870722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.870712 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.871781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.871749 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:32.871886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.871849 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.871944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.871887 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:32.872007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.871992 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz" pod "110e6a5c-b832-44b6-bf4d-fb91d24fb59a" (UID: "110e6a5c-b832-44b6-bf4d-fb91d24fb59a"). InnerVolumeSpecName "kube-api-access-pm5tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:32.972080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.972008 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.972080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.972032 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.972080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.972042 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.972080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.972053 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5tz\" (UniqueName: \"kubernetes.io/projected/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-kube-api-access-pm5tz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.972080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:32.972067 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/110e6a5c-b832-44b6-bf4d-fb91d24fb59a-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.240617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:33.240555 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz"] Oct 30 22:02:33.248633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:33.248609 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-kg4gz"] Oct 30 22:02:34.869512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:34.869482 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" path="/var/lib/kubelet/pods/110e6a5c-b832-44b6-bf4d-fb91d24fb59a/volumes" Oct 30 22:02:44.876559 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.876525 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr"] Oct 30 22:02:44.877041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.877027 5607 cpu_manager.go:414] "containerMap: removing container" podUID="0e9a25bf-6af5-4690-b0ed-48d5522566b2" containerName="revoke-keys" Oct 30 22:02:44.877094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.877042 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9a25bf-6af5-4690-b0ed-48d5522566b2" containerName="revoke-keys" Oct 30 22:02:44.877094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.877075 5607 cpu_manager.go:414] "containerMap: removing container" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" Oct 30 22:02:44.877094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.877080 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" Oct 30 22:02:44.877188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.877147 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9a25bf-6af5-4690-b0ed-48d5522566b2" containerName="revoke-keys" Oct 30 22:02:44.877188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.877161 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="110e6a5c-b832-44b6-bf4d-fb91d24fb59a" containerName="istio-proxy" Oct 30 22:02:44.880409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.880393 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:44.932783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.932753 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:44.976652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.976630 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:02:44.978683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.978663 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr"] Oct 30 22:02:44.982152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.982134 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:44.982217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.982173 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:44.982217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:44.982208 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkg4\" (UniqueName: \"kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.083570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.083546 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkg4\" (UniqueName: \"kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.083672 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.083660 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.083714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.083692 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.084488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.084425 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.086283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.086262 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.141984 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.141905 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkg4\" (UniqueName: \"kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4\") pod \"maas-api-revoke-sub-3002d34b-8pnzr\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.194277 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.194252 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:45.562127 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.562102 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr"] Oct 30 22:02:45.563799 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:02:45.563766 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc0c432_bf3d_45d0_8c06_8da3bb4fb587.slice/crio-1527a24cd7a4414c8503f1182efd278bb3f86e0426190c521c03f4e902243540 WatchSource:0}: Error finding container 1527a24cd7a4414c8503f1182efd278bb3f86e0426190c521c03f4e902243540: Status 404 returned error can't find the container with id 1527a24cd7a4414c8503f1182efd278bb3f86e0426190c521c03f4e902243540 Oct 30 22:02:45.786617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.786582 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz"] Oct 30 22:02:45.794832 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.794807 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.848530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.848493 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz"] Oct 30 22:02:45.891391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.891354 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"default-dockercfg-dqqb4\"" Oct 30 22:02:45.891797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.891411 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"kube-root-ca.crt\"" Oct 30 22:02:45.891797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.891428 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"premium-w1-e65019-kserve-self-signed-certs\"" Oct 30 22:02:45.891797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.891472 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"openshift-service-ca.crt\"" Oct 30 22:02:45.893082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.893064 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n24r\" (UniqueName: \"kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.893165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.893131 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.893241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.893211 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.893292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.893259 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.893292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.893285 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.893479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.893462 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.900050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.900022 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" event={"ID":"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587","Type":"ContainerStarted","Data":"1527a24cd7a4414c8503f1182efd278bb3f86e0426190c521c03f4e902243540"} Oct 30 22:02:45.994669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.994632 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.994875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.994681 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.994875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.994719 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.994875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.994749 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.994875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.994862 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.995141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.994891 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n24r\" (UniqueName: \"kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.995141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.995101 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.995251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.995150 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.995353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.995336 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.997057 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.997040 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:45.997599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:45.997576 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:46.057019 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.056992 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n24r\" (UniqueName: \"kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r\") pod \"premium-w1-e65019-kserve-56d658b4f7-7m7zz\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:46.130628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.130549 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:46.531821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.531793 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz"] Oct 30 22:02:46.533646 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:02:46.533615 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12f8222d_6aac_4906_aba3_54f1ecb79165.slice/crio-2a1333effc045b535c7289029e1a701654b0740767903b009756da5cb10be32a WatchSource:0}: Error finding container 2a1333effc045b535c7289029e1a701654b0740767903b009756da5cb10be32a: Status 404 returned error can't find the container with id 2a1333effc045b535c7289029e1a701654b0740767903b009756da5cb10be32a Oct 30 22:02:46.907401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.907318 5607 generic.go:356] "Generic (PLEG): container finished" podID="bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" containerID="e3221fdc0bd06b7038bb3a383bb68fc4b824da42b2741f8cfaf4aadc1193d935" exitCode=0 Oct 30 22:02:46.907401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.907367 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" event={"ID":"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587","Type":"ContainerDied","Data":"e3221fdc0bd06b7038bb3a383bb68fc4b824da42b2741f8cfaf4aadc1193d935"} Oct 30 22:02:46.909432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.909409 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" event={"ID":"12f8222d-6aac-4906-aba3-54f1ecb79165","Type":"ContainerStarted","Data":"120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59"} Oct 30 22:02:46.909570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:46.909438 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" event={"ID":"12f8222d-6aac-4906-aba3-54f1ecb79165","Type":"ContainerStarted","Data":"2a1333effc045b535c7289029e1a701654b0740767903b009756da5cb10be32a"} Oct 30 22:02:47.129837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:47.129787 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podStartSLOduration=2.129770225 podStartE2EDuration="2.129770225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:47.129481986 +0000 UTC m=+946.530038109" watchObservedRunningTime="2026-10-30 22:02:47.129770225 +0000 UTC m=+946.530326337" Oct 30 22:02:47.131434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:47.131418 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:48.058233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.058208 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:48.221136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.221108 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca\") pod \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " Oct 30 22:02:48.221323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.221189 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4\" (UniqueName: \"kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4\") pod \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " Oct 30 22:02:48.221323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.221281 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca\") pod \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\" (UID: \"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587\") " Oct 30 22:02:48.221627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.221598 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca" pod "bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" (UID: "bdc0c432-bf3d-45d0-8c06-8da3bb4fb587"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:48.223276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.223255 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca" pod "bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" (UID: "bdc0c432-bf3d-45d0-8c06-8da3bb4fb587"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:48.223462 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.223442 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4" pod "bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" (UID: "bdc0c432-bf3d-45d0-8c06-8da3bb4fb587"). InnerVolumeSpecName "kube-api-access-qvkg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:48.322687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.322654 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:48.322687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.322682 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvkg4\" (UniqueName: \"kubernetes.io/projected/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-kube-api-access-qvkg4\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:48.322846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.322696 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:48.922089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.922064 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" Oct 30 22:02:48.922213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.922087 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr" event={"ID":"bdc0c432-bf3d-45d0-8c06-8da3bb4fb587","Type":"ContainerDied","Data":"1527a24cd7a4414c8503f1182efd278bb3f86e0426190c521c03f4e902243540"} Oct 30 22:02:48.922213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:48.922139 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1527a24cd7a4414c8503f1182efd278bb3f86e0426190c521c03f4e902243540" Oct 30 22:02:52.863071 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:02:52.863037 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:02:56.131578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:56.131537 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:56.144363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:56.144337 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:02:56.984054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:02:56.984023 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:03:00.124405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.124371 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw"] Oct 30 22:03:00.124900 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.124886 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" containerName="revoke-keys" Oct 30 22:03:00.124944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.124901 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" containerName="revoke-keys" Oct 30 22:03:00.125050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.125039 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" containerName="revoke-keys" Oct 30 22:03:00.128682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.128665 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.227173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.227154 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:00.227301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.227190 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:03:00.229259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.229239 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw"] Oct 30 22:03:00.238672 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.238645 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.238821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.238788 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkg2v\" (UniqueName: \"kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.238898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.238871 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.339579 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.339554 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.339701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.339613 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.339701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.339674 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkg2v\" (UniqueName: \"kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.340191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.340172 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.341815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.341798 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.397454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.397397 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkg2v\" (UniqueName: \"kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v\") pod \"maas-api-revoke-sub-8d05022b-f2tzw\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.442173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.442153 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:00.813062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.813043 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw"] Oct 30 22:03:00.814841 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:00.814816 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1143ae_4dc6_4156_b1ea_52d1dfb547c9.slice/crio-6220b60aee71da5e93a538246c2e54961b170b3bc18b01a1b5dded78c477212f WatchSource:0}: Error finding container 6220b60aee71da5e93a538246c2e54961b170b3bc18b01a1b5dded78c477212f: Status 404 returned error can't find the container with id 6220b60aee71da5e93a538246c2e54961b170b3bc18b01a1b5dded78c477212f Oct 30 22:03:00.999580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:00.999547 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" event={"ID":"af1143ae-4dc6-4156-b1ea-52d1dfb547c9","Type":"ContainerStarted","Data":"6220b60aee71da5e93a538246c2e54961b170b3bc18b01a1b5dded78c477212f"} Oct 30 22:03:02.007055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:02.007022 5607 generic.go:356] "Generic (PLEG): container finished" podID="af1143ae-4dc6-4156-b1ea-52d1dfb547c9" containerID="38654876f2d1cc04ffa2d0b33571197491bc5d70dbcd4d59e7af0a038f5f5379" exitCode=0 Oct 30 22:03:02.007450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:02.007084 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" event={"ID":"af1143ae-4dc6-4156-b1ea-52d1dfb547c9","Type":"ContainerDied","Data":"38654876f2d1cc04ffa2d0b33571197491bc5d70dbcd4d59e7af0a038f5f5379"} Oct 30 22:03:03.167640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.167615 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:03.265596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.265533 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v\" (UniqueName: \"kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v\") pod \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " Oct 30 22:03:03.265596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.265574 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca\") pod \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " Oct 30 22:03:03.265886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.265620 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca\") pod \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\" (UID: \"af1143ae-4dc6-4156-b1ea-52d1dfb547c9\") " Oct 30 22:03:03.265942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.265925 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca" pod "af1143ae-4dc6-4156-b1ea-52d1dfb547c9" (UID: "af1143ae-4dc6-4156-b1ea-52d1dfb547c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:03.266144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.266127 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.267786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.267763 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca" pod "af1143ae-4dc6-4156-b1ea-52d1dfb547c9" (UID: "af1143ae-4dc6-4156-b1ea-52d1dfb547c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:03.267867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.267780 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v" pod "af1143ae-4dc6-4156-b1ea-52d1dfb547c9" (UID: "af1143ae-4dc6-4156-b1ea-52d1dfb547c9"). InnerVolumeSpecName "kube-api-access-hkg2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:03.367547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.367520 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkg2v\" (UniqueName: \"kubernetes.io/projected/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-kube-api-access-hkg2v\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:03.367547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:03.367543 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1143ae-4dc6-4156-b1ea-52d1dfb547c9-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.019478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:04.019449 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" Oct 30 22:03:04.019640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:04.019449 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw" event={"ID":"af1143ae-4dc6-4156-b1ea-52d1dfb547c9","Type":"ContainerDied","Data":"6220b60aee71da5e93a538246c2e54961b170b3bc18b01a1b5dded78c477212f"} Oct 30 22:03:04.019640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:04.019566 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6220b60aee71da5e93a538246c2e54961b170b3bc18b01a1b5dded78c477212f" Oct 30 22:03:15.572346 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.572318 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f"] Oct 30 22:03:15.572841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.572828 5607 cpu_manager.go:414] "containerMap: removing container" podUID="af1143ae-4dc6-4156-b1ea-52d1dfb547c9" containerName="revoke-keys" Oct 30 22:03:15.572884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.572842 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1143ae-4dc6-4156-b1ea-52d1dfb547c9" containerName="revoke-keys" Oct 30 22:03:15.572947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.572938 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1143ae-4dc6-4156-b1ea-52d1dfb547c9" containerName="revoke-keys" Oct 30 22:03:15.576097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.576076 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.630666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.630643 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f"] Oct 30 22:03:15.672537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.672516 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:15.672650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.672576 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:03:15.689716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.689691 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fpb4\" (UniqueName: \"kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.689861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.689722 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.689935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.689866 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.791048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.791025 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.791210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.791122 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fpb4\" (UniqueName: \"kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.791210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.791145 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.791744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.791721 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.793506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.793482 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.849001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.848939 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fpb4\" (UniqueName: \"kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4\") pod \"maas-api-revoke-sub-e2529ef5-9d75f\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:15.888690 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:15.888674 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:16.253150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:16.253122 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f"] Oct 30 22:03:16.254647 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:16.254609 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb899f2_bba8_460b_b35f_66741b879c16.slice/crio-1e639c3fe8cd7483037d2f0c9262d7d7fe44a554fd32e604b6385982e661d87c WatchSource:0}: Error finding container 1e639c3fe8cd7483037d2f0c9262d7d7fe44a554fd32e604b6385982e661d87c: Status 404 returned error can't find the container with id 1e639c3fe8cd7483037d2f0c9262d7d7fe44a554fd32e604b6385982e661d87c Oct 30 22:03:17.096485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:17.096450 5607 generic.go:356] "Generic (PLEG): container finished" podID="0bb899f2-bba8-460b-b35f-66741b879c16" containerID="24c0f579ff658a795f15a6fe688372b31834d30681bd6c62d17e8a73367e223d" exitCode=0 Oct 30 22:03:17.096800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:17.096538 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" event={"ID":"0bb899f2-bba8-460b-b35f-66741b879c16","Type":"ContainerDied","Data":"24c0f579ff658a795f15a6fe688372b31834d30681bd6c62d17e8a73367e223d"} Oct 30 22:03:17.096800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:17.096588 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" event={"ID":"0bb899f2-bba8-460b-b35f-66741b879c16","Type":"ContainerStarted","Data":"1e639c3fe8cd7483037d2f0c9262d7d7fe44a554fd32e604b6385982e661d87c"} Oct 30 22:03:18.245334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.245312 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:18.317109 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.317085 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca\") pod \"0bb899f2-bba8-460b-b35f-66741b879c16\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " Oct 30 22:03:18.317251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.317214 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4\" (UniqueName: \"kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4\") pod \"0bb899f2-bba8-460b-b35f-66741b879c16\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " Oct 30 22:03:18.317251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.317246 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca\") pod \"0bb899f2-bba8-460b-b35f-66741b879c16\" (UID: \"0bb899f2-bba8-460b-b35f-66741b879c16\") " Oct 30 22:03:18.317673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.317637 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca" pod "0bb899f2-bba8-460b-b35f-66741b879c16" (UID: "0bb899f2-bba8-460b-b35f-66741b879c16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:18.319234 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.319214 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca" pod "0bb899f2-bba8-460b-b35f-66741b879c16" (UID: "0bb899f2-bba8-460b-b35f-66741b879c16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:18.319337 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.319245 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4" pod "0bb899f2-bba8-460b-b35f-66741b879c16" (UID: "0bb899f2-bba8-460b-b35f-66741b879c16"). InnerVolumeSpecName "kube-api-access-9fpb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:18.418481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.418421 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fpb4\" (UniqueName: \"kubernetes.io/projected/0bb899f2-bba8-460b-b35f-66741b879c16-kube-api-access-9fpb4\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:18.418481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.418449 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:18.418481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:18.418459 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb899f2-bba8-460b-b35f-66741b879c16-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:19.108478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:19.108445 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" event={"ID":"0bb899f2-bba8-460b-b35f-66741b879c16","Type":"ContainerDied","Data":"1e639c3fe8cd7483037d2f0c9262d7d7fe44a554fd32e604b6385982e661d87c"} Oct 30 22:03:19.108644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:19.108499 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e639c3fe8cd7483037d2f0c9262d7d7fe44a554fd32e604b6385982e661d87c" Oct 30 22:03:19.108644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:19.108457 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f" Oct 30 22:03:21.253980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.253932 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr"] Oct 30 22:03:21.254659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.254637 5607 cpu_manager.go:414] "containerMap: removing container" podUID="0bb899f2-bba8-460b-b35f-66741b879c16" containerName="revoke-keys" Oct 30 22:03:21.254760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.254659 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb899f2-bba8-460b-b35f-66741b879c16" containerName="revoke-keys" Oct 30 22:03:21.254818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.254804 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb899f2-bba8-460b-b35f-66741b879c16" containerName="revoke-keys" Oct 30 22:03:21.257765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.257747 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.307171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.307151 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:21.346405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.346381 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.346528 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.346451 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.346582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.346522 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqw52\" (UniqueName: \"kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.353931 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.353910 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:03:21.356237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.356219 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr"] Oct 30 22:03:21.448095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.448075 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.448217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.448102 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqw52\" (UniqueName: \"kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.448217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.448200 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.448833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.448809 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.450580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.450562 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.507137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.507068 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqw52\" (UniqueName: \"kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52\") pod \"maas-api-revoke-sub-169d8e08-qtrqr\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.569926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.569898 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:21.936432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:21.936406 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr"] Oct 30 22:03:21.938192 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:21.938165 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6051c04_fcce_4b4b_b0d6_8d10f3620985.slice/crio-a3ed1b1b8c4131c47a054680bd629b77f865e246dcf27aaf54b6fd7a731fdf8a WatchSource:0}: Error finding container a3ed1b1b8c4131c47a054680bd629b77f865e246dcf27aaf54b6fd7a731fdf8a: Status 404 returned error can't find the container with id a3ed1b1b8c4131c47a054680bd629b77f865e246dcf27aaf54b6fd7a731fdf8a Oct 30 22:03:22.129537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:22.129507 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" event={"ID":"d6051c04-fcce-4b4b-b0d6-8d10f3620985","Type":"ContainerStarted","Data":"a3ed1b1b8c4131c47a054680bd629b77f865e246dcf27aaf54b6fd7a731fdf8a"} Oct 30 22:03:22.862400 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:03:22.862355 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:03:23.136146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:23.136109 5607 generic.go:356] "Generic (PLEG): container finished" podID="d6051c04-fcce-4b4b-b0d6-8d10f3620985" containerID="e07235c353950345ffaa50c02d31756de6db68a55fcd04dc14e83372a28b4e41" exitCode=0 Oct 30 22:03:23.136319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:23.136198 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" event={"ID":"d6051c04-fcce-4b4b-b0d6-8d10f3620985","Type":"ContainerDied","Data":"e07235c353950345ffaa50c02d31756de6db68a55fcd04dc14e83372a28b4e41"} Oct 30 22:03:24.285306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.285283 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:24.381048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.381022 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca\") pod \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " Oct 30 22:03:24.381177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.381086 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca\") pod \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " Oct 30 22:03:24.381177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.381126 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52\" (UniqueName: \"kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52\") pod \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\" (UID: \"d6051c04-fcce-4b4b-b0d6-8d10f3620985\") " Oct 30 22:03:24.381501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.381471 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca" pod "d6051c04-fcce-4b4b-b0d6-8d10f3620985" (UID: "d6051c04-fcce-4b4b-b0d6-8d10f3620985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:24.383210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.383187 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca" pod "d6051c04-fcce-4b4b-b0d6-8d10f3620985" (UID: "d6051c04-fcce-4b4b-b0d6-8d10f3620985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:24.383291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.383227 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52" pod "d6051c04-fcce-4b4b-b0d6-8d10f3620985" (UID: "d6051c04-fcce-4b4b-b0d6-8d10f3620985"). InnerVolumeSpecName "kube-api-access-pqw52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:24.482653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.482629 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:24.482653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.482652 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6051c04-fcce-4b4b-b0d6-8d10f3620985-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:24.482835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:24.482663 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqw52\" (UniqueName: \"kubernetes.io/projected/d6051c04-fcce-4b4b-b0d6-8d10f3620985-kube-api-access-pqw52\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:25.149095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:25.149019 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" Oct 30 22:03:25.149095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:25.149026 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr" event={"ID":"d6051c04-fcce-4b4b-b0d6-8d10f3620985","Type":"ContainerDied","Data":"a3ed1b1b8c4131c47a054680bd629b77f865e246dcf27aaf54b6fd7a731fdf8a"} Oct 30 22:03:25.149095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:25.149076 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ed1b1b8c4131c47a054680bd629b77f865e246dcf27aaf54b6fd7a731fdf8a" Oct 30 22:03:32.065277 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.065241 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj"] Oct 30 22:03:32.065747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.065733 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d6051c04-fcce-4b4b-b0d6-8d10f3620985" containerName="revoke-keys" Oct 30 22:03:32.065790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.065748 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6051c04-fcce-4b4b-b0d6-8d10f3620985" containerName="revoke-keys" Oct 30 22:03:32.065861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.065851 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6051c04-fcce-4b4b-b0d6-8d10f3620985" containerName="revoke-keys" Oct 30 22:03:32.069175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.069161 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.124614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.124590 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj"] Oct 30 22:03:32.161536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.161515 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.161654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.161561 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.161654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.161583 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7cq2\" (UniqueName: \"kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.168217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.168196 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:03:32.168217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.168192 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:32.262644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.262617 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7cq2\" (UniqueName: \"kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.262767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.262734 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.262834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.262778 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.263377 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.263355 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.265301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.265275 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.320662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.320606 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7cq2\" (UniqueName: \"kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2\") pod \"maas-api-revoke-sub-0cfad996-lckxj\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.382337 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.382317 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:32.751819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:32.751794 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj"] Oct 30 22:03:32.753834 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:32.753809 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7c8905_7410_4af6_b7eb_192b2e2c071e.slice/crio-35b3c289458c68e10eae6901d8ec3f7204089e9c4d2aaae409babdfcef574c67 WatchSource:0}: Error finding container 35b3c289458c68e10eae6901d8ec3f7204089e9c4d2aaae409babdfcef574c67: Status 404 returned error can't find the container with id 35b3c289458c68e10eae6901d8ec3f7204089e9c4d2aaae409babdfcef574c67 Oct 30 22:03:33.197358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:33.197027 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" event={"ID":"6b7c8905-7410-4af6-b7eb-192b2e2c071e","Type":"ContainerStarted","Data":"eea665249166e5eb9bec27e7a169341f0d9476c232a2fa51405a749d151b09bb"} Oct 30 22:03:33.197358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:33.197092 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" event={"ID":"6b7c8905-7410-4af6-b7eb-192b2e2c071e","Type":"ContainerStarted","Data":"35b3c289458c68e10eae6901d8ec3f7204089e9c4d2aaae409babdfcef574c67"} Oct 30 22:03:33.308608 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:33.308527 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" podStartSLOduration=1.308513436 podStartE2EDuration="1.308513436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:33.30688392 +0000 UTC m=+992.707440043" watchObservedRunningTime="2026-10-30 22:03:33.308513436 +0000 UTC m=+992.709069557" Oct 30 22:03:34.204649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:34.204613 5607 generic.go:356] "Generic (PLEG): container finished" podID="6b7c8905-7410-4af6-b7eb-192b2e2c071e" containerID="eea665249166e5eb9bec27e7a169341f0d9476c232a2fa51405a749d151b09bb" exitCode=0 Oct 30 22:03:34.205064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:34.204691 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" event={"ID":"6b7c8905-7410-4af6-b7eb-192b2e2c071e","Type":"ContainerDied","Data":"eea665249166e5eb9bec27e7a169341f0d9476c232a2fa51405a749d151b09bb"} Oct 30 22:03:35.363916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.363892 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:35.395369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.395328 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca\") pod \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " Oct 30 22:03:35.395570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.395432 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca\") pod \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " Oct 30 22:03:35.395570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.395504 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2\" (UniqueName: \"kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2\") pod \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\" (UID: \"6b7c8905-7410-4af6-b7eb-192b2e2c071e\") " Oct 30 22:03:35.395985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.395930 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca" pod "6b7c8905-7410-4af6-b7eb-192b2e2c071e" (UID: "6b7c8905-7410-4af6-b7eb-192b2e2c071e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:35.398066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.398042 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2" pod "6b7c8905-7410-4af6-b7eb-192b2e2c071e" (UID: "6b7c8905-7410-4af6-b7eb-192b2e2c071e"). InnerVolumeSpecName "kube-api-access-m7cq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:35.398163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.398066 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca" pod "6b7c8905-7410-4af6-b7eb-192b2e2c071e" (UID: "6b7c8905-7410-4af6-b7eb-192b2e2c071e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:35.497262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.497185 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:35.497262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.497209 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7cq2\" (UniqueName: \"kubernetes.io/projected/6b7c8905-7410-4af6-b7eb-192b2e2c071e-kube-api-access-m7cq2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:35.497262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:35.497220 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7c8905-7410-4af6-b7eb-192b2e2c071e-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:36.217141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.217112 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" Oct 30 22:03:36.217141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.217133 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj" event={"ID":"6b7c8905-7410-4af6-b7eb-192b2e2c071e","Type":"ContainerDied","Data":"35b3c289458c68e10eae6901d8ec3f7204089e9c4d2aaae409babdfcef574c67"} Oct 30 22:03:36.217376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.217185 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b3c289458c68e10eae6901d8ec3f7204089e9c4d2aaae409babdfcef574c67" Oct 30 22:03:36.334001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.333945 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd"] Oct 30 22:03:36.334557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.334539 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6b7c8905-7410-4af6-b7eb-192b2e2c071e" containerName="revoke-keys" Oct 30 22:03:36.334685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.334559 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7c8905-7410-4af6-b7eb-192b2e2c071e" containerName="revoke-keys" Oct 30 22:03:36.334758 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.334735 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7c8905-7410-4af6-b7eb-192b2e2c071e" containerName="revoke-keys" Oct 30 22:03:36.338247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.338230 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.390619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.390592 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 22:03:36.390921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.390619 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 22:03:36.392782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.392764 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd"] Oct 30 22:03:36.394915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.394898 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 22:03:36.395030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.394932 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4c4542dd9101\"" Oct 30 22:03:36.395030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.394932 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 22:03:36.395030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.394967 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-594b013fa537\"" Oct 30 22:03:36.395030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.395020 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6kq7g\"" Oct 30 22:03:36.406766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.406742 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.406894 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.406875 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.406935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.406907 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.407030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.407016 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkb5g\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.507683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.507605 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.507848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.507683 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.507848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.507714 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.507848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.507801 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkb5g\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.508369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.508349 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.508533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.508506 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.510669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.510643 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.566651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.566624 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkb5g\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g\") pod \"payload-processing-external-model-5ff46798db-xzrpd\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:36.650898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:36.650868 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:37.025085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:37.025060 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd"] Oct 30 22:03:37.026864 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:37.026838 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775b5cdb_1352_49c9_8362_abf8572966a5.slice/crio-de405b9600fa697d978f0725f71af3d6af18ef6e238f3bce1bb140dacc9710e4 WatchSource:0}: Error finding container de405b9600fa697d978f0725f71af3d6af18ef6e238f3bce1bb140dacc9710e4: Status 404 returned error can't find the container with id de405b9600fa697d978f0725f71af3d6af18ef6e238f3bce1bb140dacc9710e4 Oct 30 22:03:37.227093 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:37.227052 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" event={"ID":"775b5cdb-1352-49c9-8362-abf8572966a5","Type":"ContainerStarted","Data":"e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70"} Oct 30 22:03:37.227308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:37.227126 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" event={"ID":"775b5cdb-1352-49c9-8362-abf8572966a5","Type":"ContainerStarted","Data":"de405b9600fa697d978f0725f71af3d6af18ef6e238f3bce1bb140dacc9710e4"} Oct 30 22:03:37.227308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:37.227256 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:37.339010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:37.338880 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" podStartSLOduration=1.33886212 podStartE2EDuration="1.33886212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:37.337576327 +0000 UTC m=+996.738132450" watchObservedRunningTime="2026-10-30 22:03:37.33886212 +0000 UTC m=+996.739418241" Oct 30 22:03:41.207934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.207853 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh"] Oct 30 22:03:41.211782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.211765 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.263378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.263349 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:41.310237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.310211 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:03:41.313076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.313051 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh"] Oct 30 22:03:41.362561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.362531 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.362705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.362576 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mklk\" (UniqueName: \"kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.362705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.362673 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.463583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.463516 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.463583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.463559 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mklk\" (UniqueName: \"kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.463752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.463607 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.464186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.464164 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.466154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.466130 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.522067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.522038 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mklk\" (UniqueName: \"kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk\") pod \"maas-api-revoke-sub-3adba886-7l2hh\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.525707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.525687 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:41.908758 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:41.908726 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh"] Oct 30 22:03:41.910624 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:41.910595 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f779b0f_ffc7_482f_ab07_d7348e6b8c54.slice/crio-99ad60cb67ca189319b7f46f1386b100981cdace60509369344fb5ba8762fdd2 WatchSource:0}: Error finding container 99ad60cb67ca189319b7f46f1386b100981cdace60509369344fb5ba8762fdd2: Status 404 returned error can't find the container with id 99ad60cb67ca189319b7f46f1386b100981cdace60509369344fb5ba8762fdd2 Oct 30 22:03:42.267249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:42.267212 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" event={"ID":"7f779b0f-ffc7-482f-ab07-d7348e6b8c54","Type":"ContainerStarted","Data":"ff63e36944e7177404307e9ff9a6105e8648874b6415998dc74674ebe698d2af"} Oct 30 22:03:42.267663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:42.267278 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" event={"ID":"7f779b0f-ffc7-482f-ab07-d7348e6b8c54","Type":"ContainerStarted","Data":"99ad60cb67ca189319b7f46f1386b100981cdace60509369344fb5ba8762fdd2"} Oct 30 22:03:43.275650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:43.275561 5607 generic.go:356] "Generic (PLEG): container finished" podID="7f779b0f-ffc7-482f-ab07-d7348e6b8c54" containerID="ff63e36944e7177404307e9ff9a6105e8648874b6415998dc74674ebe698d2af" exitCode=0 Oct 30 22:03:43.275650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:43.275609 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" event={"ID":"7f779b0f-ffc7-482f-ab07-d7348e6b8c54","Type":"ContainerDied","Data":"ff63e36944e7177404307e9ff9a6105e8648874b6415998dc74674ebe698d2af"} Oct 30 22:03:44.462535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.462504 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:44.602080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.601986 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk\" (UniqueName: \"kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk\") pod \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " Oct 30 22:03:44.602080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.602051 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca\") pod \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " Oct 30 22:03:44.602316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.602155 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca\") pod \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\" (UID: \"7f779b0f-ffc7-482f-ab07-d7348e6b8c54\") " Oct 30 22:03:44.603535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.603509 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca" pod "7f779b0f-ffc7-482f-ab07-d7348e6b8c54" (UID: "7f779b0f-ffc7-482f-ab07-d7348e6b8c54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:44.607976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.607920 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk" pod "7f779b0f-ffc7-482f-ab07-d7348e6b8c54" (UID: "7f779b0f-ffc7-482f-ab07-d7348e6b8c54"). InnerVolumeSpecName "kube-api-access-8mklk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:44.619478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.619454 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca" pod "7f779b0f-ffc7-482f-ab07-d7348e6b8c54" (UID: "7f779b0f-ffc7-482f-ab07-d7348e6b8c54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:44.705000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.704916 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:44.705000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.704974 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mklk\" (UniqueName: \"kubernetes.io/projected/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-kube-api-access-8mklk\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:44.705000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:44.704992 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f779b0f-ffc7-482f-ab07-d7348e6b8c54-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:45.288784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:45.288752 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" Oct 30 22:03:45.288936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:45.288800 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh" event={"ID":"7f779b0f-ffc7-482f-ab07-d7348e6b8c54","Type":"ContainerDied","Data":"99ad60cb67ca189319b7f46f1386b100981cdace60509369344fb5ba8762fdd2"} Oct 30 22:03:45.288936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:45.288847 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99ad60cb67ca189319b7f46f1386b100981cdace60509369344fb5ba8762fdd2" Oct 30 22:03:48.237612 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:48.237576 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:03:52.863842 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:03:52.863803 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:03:53.539802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.539770 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd"] Oct 30 22:03:53.540274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.540262 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7f779b0f-ffc7-482f-ab07-d7348e6b8c54" containerName="revoke-keys" Oct 30 22:03:53.540327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.540275 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f779b0f-ffc7-482f-ab07-d7348e6b8c54" containerName="revoke-keys" Oct 30 22:03:53.540388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.540379 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f779b0f-ffc7-482f-ab07-d7348e6b8c54" containerName="revoke-keys" Oct 30 22:03:53.543876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.543861 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.643088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.643059 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"default-dockercfg-v8p5t\"" Oct 30 22:03:53.643257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.643097 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"openshift-service-ca.crt\"" Oct 30 22:03:53.643257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.643114 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"e2e-conflict-a-d3c7fa24-kserve-self-signed-certs\"" Oct 30 22:03:53.643257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.643201 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"kube-root-ca.crt\"" Oct 30 22:03:53.645669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.645651 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd"] Oct 30 22:03:53.684126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.684096 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.684249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.684145 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfwg\" (UniqueName: \"kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.684249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.684239 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.684322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.684268 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.684360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.684332 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.684391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.684375 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785180 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785212 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785236 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785265 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785398 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785445 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfwg\" (UniqueName: \"kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785638 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785690 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.785833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.785813 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.787705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.787684 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.788075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.788055 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.845496 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.845438 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfwg\" (UniqueName: \"kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg\") pod \"e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:53.855313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:53.855294 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:54.235355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:54.235322 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd"] Oct 30 22:03:54.236577 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:54.236553 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod535e6f8e_5e47_461c_b469_ed0ae0bf6388.slice/crio-45a2366b651bdd9985a52eca0b4f13a8d4d5b84315ddcc79388532a8fed54be8 WatchSource:0}: Error finding container 45a2366b651bdd9985a52eca0b4f13a8d4d5b84315ddcc79388532a8fed54be8: Status 404 returned error can't find the container with id 45a2366b651bdd9985a52eca0b4f13a8d4d5b84315ddcc79388532a8fed54be8 Oct 30 22:03:54.344504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:54.344474 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" event={"ID":"535e6f8e-5e47-461c-b469-ed0ae0bf6388","Type":"ContainerStarted","Data":"def6a689b759b30beff538e6d069994f61e4965c3a83ee7b2db5f65463a44f55"} Oct 30 22:03:54.344638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:54.344520 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" event={"ID":"535e6f8e-5e47-461c-b469-ed0ae0bf6388","Type":"ContainerStarted","Data":"45a2366b651bdd9985a52eca0b4f13a8d4d5b84315ddcc79388532a8fed54be8"} Oct 30 22:03:54.857070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:54.857039 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:03:58.163067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.163012 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" podStartSLOduration=5.162998748 podStartE2EDuration="5.162998748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:54.457303914 +0000 UTC m=+1013.857860046" watchObservedRunningTime="2026-10-30 22:03:58.162998748 +0000 UTC m=+1017.563554920" Oct 30 22:03:58.163829 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.163811 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t"] Oct 30 22:03:58.167929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.167906 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.210370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.210351 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h"] Oct 30 22:03:58.213968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.213936 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.269153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.269134 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t"] Oct 30 22:03:58.315783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.315759 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h"] Oct 30 22:03:58.329917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.329895 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.329924 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.329971 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330086 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330117 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330171 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330195 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330245 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330213 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330245 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330229 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330333 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330357 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p96z\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330379 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330402 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330469 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwc6\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330513 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330589 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.330646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330624 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.330686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.330661 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.431475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431390 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.431475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431438 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.431475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431463 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.431715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431506 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.431715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431537 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.431715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431564 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.431850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431736 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.431850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431785 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.431850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431817 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.432049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431862 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431898 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431922 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p96z\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.431988 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432029 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432066 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwc6\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.432474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432149 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.432474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432180 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432234 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.432474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432296 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.432474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432356 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432795 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432614 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.432856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.432804 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.433037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.433013 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.433220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.433200 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.433320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.433070 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.433320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.433292 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.433554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.433527 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.433816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.433794 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.434320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.434298 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.435391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.435023 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.446635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.446202 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.449306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.449287 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.490342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.490306 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.491208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.491188 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwc6\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6\") pod \"maas-default-gateway-openshift-default-6f54779779-8rl4t\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.491422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.491401 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.491640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.491619 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p96z\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z\") pod \"maas-default-gateway-openshift-default-6f54779779-h2z4h\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.526606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.526588 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:58.782250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.782223 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:03:58.946351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.946323 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h"] Oct 30 22:03:58.947173 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:58.947150 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a2e457_26de_4744_b8a7_39f952cb4cae.slice/crio-ccb37d5ff064e0034cb9e924902b45fdef7951b6106cfbb09b2c04d768d93c0f WatchSource:0}: Error finding container ccb37d5ff064e0034cb9e924902b45fdef7951b6106cfbb09b2c04d768d93c0f: Status 404 returned error can't find the container with id ccb37d5ff064e0034cb9e924902b45fdef7951b6106cfbb09b2c04d768d93c0f Oct 30 22:03:58.949568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.949534 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:58.949642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.949602 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:58.949642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:58.949632 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.154281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.154253 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t"] Oct 30 22:03:59.155755 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:03:59.155729 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6274436d_a97f_4298_9edf_1b1a68fc1332.slice/crio-2e9f6ba877badb991611019e1296856ffe08a30d6af570e0642ebccf17d75814 WatchSource:0}: Error finding container 2e9f6ba877badb991611019e1296856ffe08a30d6af570e0642ebccf17d75814: Status 404 returned error can't find the container with id 2e9f6ba877badb991611019e1296856ffe08a30d6af570e0642ebccf17d75814 Oct 30 22:03:59.158727 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.158680 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.158820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.158788 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.158881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.158835 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.375542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.375505 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" event={"ID":"6274436d-a97f-4298-9edf-1b1a68fc1332","Type":"ContainerStarted","Data":"66cdad24981357135db4e34a60262935921293bf40d00360a66cd7ffd2f0c8e8"} Oct 30 22:03:59.376045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.375573 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" event={"ID":"6274436d-a97f-4298-9edf-1b1a68fc1332","Type":"ContainerStarted","Data":"2e9f6ba877badb991611019e1296856ffe08a30d6af570e0642ebccf17d75814"} Oct 30 22:03:59.377354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.377329 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" event={"ID":"47a2e457-26de-4744-b8a7-39f952cb4cae","Type":"ContainerStarted","Data":"e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9"} Oct 30 22:03:59.377456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.377375 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" event={"ID":"47a2e457-26de-4744-b8a7-39f952cb4cae","Type":"ContainerStarted","Data":"ccb37d5ff064e0034cb9e924902b45fdef7951b6106cfbb09b2c04d768d93c0f"} Oct 30 22:03:59.492615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.492546 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podStartSLOduration=1.492527169 podStartE2EDuration="1.492527169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:59.490031523 +0000 UTC m=+1018.890587645" watchObservedRunningTime="2026-10-30 22:03:59.492527169 +0000 UTC m=+1018.893083291" Oct 30 22:03:59.526784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.526757 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:03:59.782467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:03:59.782371 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:04:00.530983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:00.529281 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:00.530983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:00.529342 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:00.784426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:00.784347 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:00.784426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:00.784407 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:01.562895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:01.562755 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:01.563654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:01.563617 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:01.785860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:01.785803 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:01.786087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:01.785885 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:02.536979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:02.532049 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:02.536979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:02.532107 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:02.783623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:02.783589 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:02.784038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:02.783644 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:03.545024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.537321 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.545024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.537383 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:03.783452 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.783420 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.783625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.783468 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:03.862037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.861521 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:04:03.876449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.876422 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:04:03.992918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:03.992863 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podStartSLOduration=5.992849058 podStartE2EDuration="5.992849058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:59.606564595 +0000 UTC m=+1019.007120722" watchObservedRunningTime="2026-10-30 22:04:03.992849058 +0000 UTC m=+1023.393405179" Oct 30 22:04:04.462822 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:04.462796 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:04:04.549406 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:04.549370 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.549626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:04.549432 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:04.783863 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:04.783779 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.783863 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:04.783833 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:05.527149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:05.527111 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.527680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:05.527174 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:05.783438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:05.783362 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.783567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:05.783428 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:06.528182 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:06.528144 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:04:06.528654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:06.528209 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:04:06.782933 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:06.782853 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:04:06.782933 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:06.782915 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 22:04:08.375700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.375667 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th"] Oct 30 22:04:08.381361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.381342 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.434175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.434152 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.434322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.434181 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.434322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.434198 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz268\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.434463 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.434346 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.459515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.459480 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th"] Oct 30 22:04:08.527257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.527217 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:08.527455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.527274 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:08.535774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.535746 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.535774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.535776 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.536007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.535795 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz268\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.536007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.535846 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.536625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.536600 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.536736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.536689 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.538403 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.538381 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.594016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.593995 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz268\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268\") pod \"payload-processing-external-model-7474dfb69b-gj7th\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.695473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.695450 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:08.786138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.785832 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:08.786138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:08.785904 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:09.086628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.086599 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th"] Oct 30 22:04:09.088648 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:04:09.088614 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9d3ffd_6639_4b2c_9e8f_718948a0980f.slice/crio-c498ba0c6be82ad60e03ca2792a36e67293328dc0c31e96dc5e8c23ededaf786 WatchSource:0}: Error finding container c498ba0c6be82ad60e03ca2792a36e67293328dc0c31e96dc5e8c23ededaf786: Status 404 returned error can't find the container with id c498ba0c6be82ad60e03ca2792a36e67293328dc0c31e96dc5e8c23ededaf786 Oct 30 22:04:09.486588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.486557 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" event={"ID":"3d9d3ffd-6639-4b2c-9e8f-718948a0980f","Type":"ContainerStarted","Data":"e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9"} Oct 30 22:04:09.487018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.486612 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" event={"ID":"3d9d3ffd-6639-4b2c-9e8f-718948a0980f","Type":"ContainerStarted","Data":"c498ba0c6be82ad60e03ca2792a36e67293328dc0c31e96dc5e8c23ededaf786"} Oct 30 22:04:09.487018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.486645 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:04:09.569085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.569060 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:09.569250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.569101 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:09.597799 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.597748 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podStartSLOduration=1.597730906 podStartE2EDuration="1.597730906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:09.597219716 +0000 UTC m=+1028.997775840" watchObservedRunningTime="2026-10-30 22:04:09.597730906 +0000 UTC m=+1028.998287029" Oct 30 22:04:09.828132 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.828041 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:09.828132 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:09.828107 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:10.370338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:10.370305 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:04:10.370910 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:10.370886 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:04:10.371804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:10.371783 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:04:10.560002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:10.559972 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:04:10.560898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:10.560873 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:04:10.561872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:10.561842 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:04:20.509979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:20.509057 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:22.863074 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:04:22.863026 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:04:30.494647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:30.494607 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:37.507421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.507391 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg"] Oct 30 22:04:37.512443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.512427 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.618928 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.618892 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"distinct-w4-e524ce-kserve-self-signed-certs\"" Oct 30 22:04:37.621588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.621569 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg"] Oct 30 22:04:37.650836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.650809 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.650972 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.650875 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.651019 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.650994 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.651055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.651035 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjqx\" (UniqueName: \"kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.651142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.651126 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.651225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.651153 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752278 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752314 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752373 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752426 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752465 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752498 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjqx\" (UniqueName: \"kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752772 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752797 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.752911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.752862 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.755048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.755029 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.755210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.755192 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.814910 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.814842 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjqx\" (UniqueName: \"kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx\") pod \"distinct-w4-e524ce-kserve-894955b77-v5qvg\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:37.824691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:37.824670 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:38.194700 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:04:38.194671 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0a7fe83_aac9_44eb_847a_d3246cd0a1de.slice/crio-2a1013ff89d97faaa29119c3864978f3b30fa0d61caefd9498ab8105c35182ae WatchSource:0}: Error finding container 2a1013ff89d97faaa29119c3864978f3b30fa0d61caefd9498ab8105c35182ae: Status 404 returned error can't find the container with id 2a1013ff89d97faaa29119c3864978f3b30fa0d61caefd9498ab8105c35182ae Oct 30 22:04:38.197373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:38.197352 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg"] Oct 30 22:04:38.678333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:38.678252 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" event={"ID":"c0a7fe83-aac9-44eb-847a-d3246cd0a1de","Type":"ContainerStarted","Data":"b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c"} Oct 30 22:04:38.678333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:38.678299 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" event={"ID":"c0a7fe83-aac9-44eb-847a-d3246cd0a1de","Type":"ContainerStarted","Data":"2a1013ff89d97faaa29119c3864978f3b30fa0d61caefd9498ab8105c35182ae"} Oct 30 22:04:38.798436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:38.798390 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podStartSLOduration=1.79837709 podStartE2EDuration="1.79837709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:38.796665645 +0000 UTC m=+1058.197221765" watchObservedRunningTime="2026-10-30 22:04:38.79837709 +0000 UTC m=+1058.198933252" Oct 30 22:04:38.825631 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:38.825607 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:40.494927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:40.494893 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:43.726721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.726691 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q"] Oct 30 22:04:43.730895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.730880 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:43.836085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.836055 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q"] Oct 30 22:04:43.837428 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.837413 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"distinct-w1-e65019-kserve-self-signed-certs\"" Oct 30 22:04:43.919443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.919408 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:43.919620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.919542 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:43.919620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.919583 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:43.919620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.919614 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx447\" (UniqueName: \"kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:43.919761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.919632 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:43.919799 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:43.919754 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021216 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021271 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021313 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx447\" (UniqueName: \"kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021330 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021350 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021496 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021638 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021785 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021765 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.021886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.021865 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.023774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.023754 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.024044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.024026 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.083507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.083479 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx447\" (UniqueName: \"kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447\") pod \"distinct-w1-e65019-kserve-5d854f576d-sb25q\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.343741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.343657 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:44.718796 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:04:44.718766 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98f3e9be_0d98_419d_8bb0_8309858d897a.slice/crio-94b400a02a2f3fbd935ec3b4726e478b566c61d6dc2bc073514b884a14f575c2 WatchSource:0}: Error finding container 94b400a02a2f3fbd935ec3b4726e478b566c61d6dc2bc073514b884a14f575c2: Status 404 returned error can't find the container with id 94b400a02a2f3fbd935ec3b4726e478b566c61d6dc2bc073514b884a14f575c2 Oct 30 22:04:44.722382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:44.722363 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q"] Oct 30 22:04:45.722680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:45.722644 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" event={"ID":"98f3e9be-0d98-419d-8bb0-8309858d897a","Type":"ContainerStarted","Data":"7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0"} Oct 30 22:04:45.723119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:45.722692 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" event={"ID":"98f3e9be-0d98-419d-8bb0-8309858d897a","Type":"ContainerStarted","Data":"94b400a02a2f3fbd935ec3b4726e478b566c61d6dc2bc073514b884a14f575c2"} Oct 30 22:04:45.844761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:45.844694 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" podStartSLOduration=2.844680381 podStartE2EDuration="2.844680381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:45.842647162 +0000 UTC m=+1065.243203282" watchObservedRunningTime="2026-10-30 22:04:45.844680381 +0000 UTC m=+1065.245236552" Oct 30 22:04:46.344319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:46.344288 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:47.825870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:47.825839 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:47.838724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:47.838699 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:48.754503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:48.754478 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:04:50.494909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:50.494876 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:52.863051 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:04:52.863017 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:04:54.343828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:54.343799 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:54.356038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:54.356015 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:04:54.798906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:04:54.798884 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:05:00.494140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:00.494099 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:10.494552 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:10.494461 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:10.999031 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:10.998998 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4c4542dd9101: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:10.999031 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:10.999035 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:10.999379 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:10.999361 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:11.499338467 +0000 UTC m=+1090.899894582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:11.504015 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:11.503989 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4c4542dd9101: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:11.504015 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:11.504013 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:11.504440 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:11.504069 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:12.504054826 +0000 UTC m=+1091.904610939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:12.515905 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:12.515871 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4c4542dd9101: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:12.515905 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:12.515900 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:12.516441 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:12.515988 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:14.515946423 +0000 UTC m=+1093.916502522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:14.537482 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:14.537451 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4c4542dd9101: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:14.537482 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:14.537477 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:14.537917 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:14.537537 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:18.537521985 +0000 UTC m=+1097.938078085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-4c4542dd9101" not found Oct 30 22:05:18.585073 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:18.585042 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-594b013fa537: secret "agc-cred-594b013fa537" not found Oct 30 22:05:18.585073 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:18.585066 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-594b013fa537" not found Oct 30 22:05:18.585499 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:18.585128 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:26.585113377 +0000 UTC m=+1105.985669494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-594b013fa537" not found Oct 30 22:05:20.241798 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.241762 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th"] Oct 30 22:05:20.243013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.242985 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:20.277696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.277669 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-9ld8d"] Oct 30 22:05:20.281916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.281899 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.357707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.357682 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-9ld8d"] Oct 30 22:05:20.405748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.405719 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.405882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.405846 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.405943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.405925 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.406037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.406025 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wg5k\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.506891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.506830 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.507061 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.506892 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wg5k\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.507061 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.506993 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.507187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.507068 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.507681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.507656 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.507765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.507750 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.509607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.509583 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.570333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.570303 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wg5k\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k\") pod \"payload-processing-external-model-6844c89498-9ld8d\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.595741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.595720 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:20.965604 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:05:20.965572 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359ab5c8_014e_421c_b13d_ce3fbcb37506.slice/crio-a452637c594bfc19c7c3632ee58ceef57fdc5caac6e97926a9061c1147f921c9 WatchSource:0}: Error finding container a452637c594bfc19c7c3632ee58ceef57fdc5caac6e97926a9061c1147f921c9: Status 404 returned error can't find the container with id a452637c594bfc19c7c3632ee58ceef57fdc5caac6e97926a9061c1147f921c9 Oct 30 22:05:20.969594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:20.969573 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-9ld8d"] Oct 30 22:05:21.956242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:21.956212 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" event={"ID":"359ab5c8-014e-421c-b13d-ce3fbcb37506","Type":"ContainerStarted","Data":"e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40"} Oct 30 22:05:21.956627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:21.956265 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" event={"ID":"359ab5c8-014e-421c-b13d-ce3fbcb37506","Type":"ContainerStarted","Data":"a452637c594bfc19c7c3632ee58ceef57fdc5caac6e97926a9061c1147f921c9"} Oct 30 22:05:21.956627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:21.956295 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:05:22.078838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:22.078789 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podStartSLOduration=2.078775643 podStartE2EDuration="2.078775643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:22.076585094 +0000 UTC m=+1101.477141217" watchObservedRunningTime="2026-10-30 22:05:22.078775643 +0000 UTC m=+1101.479331763" Oct 30 22:05:22.887015 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:22.886655 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:05:25.242974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.242213 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" containerID="cri-o://e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9" gracePeriod=25 Oct 30 22:05:25.527178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.527152 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:05:25.592366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592339 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume\") pod \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " Oct 30 22:05:25.592512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592378 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume\") pod \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " Oct 30 22:05:25.592563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592526 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268\") pod \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " Oct 30 22:05:25.592563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592546 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials\") pod \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\" (UID: \"3d9d3ffd-6639-4b2c-9e8f-718948a0980f\") " Oct 30 22:05:25.592773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592746 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume" pod "3d9d3ffd-6639-4b2c-9e8f-718948a0980f" (UID: "3d9d3ffd-6639-4b2c-9e8f-718948a0980f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:25.592881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592783 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume" pod "3d9d3ffd-6639-4b2c-9e8f-718948a0980f" (UID: "3d9d3ffd-6639-4b2c-9e8f-718948a0980f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:25.593003 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.592987 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:25.593060 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.593009 5607 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-routing-overlay-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:25.594987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.594927 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials" pod "3d9d3ffd-6639-4b2c-9e8f-718948a0980f" (UID: "3d9d3ffd-6639-4b2c-9e8f-718948a0980f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:25.594987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.594980 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268" pod "3d9d3ffd-6639-4b2c-9e8f-718948a0980f" (UID: "3d9d3ffd-6639-4b2c-9e8f-718948a0980f"). InnerVolumeSpecName "kube-api-access-xz268". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:25.693810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.693785 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz268\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-kube-api-access-xz268\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:25.693810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.693812 5607 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d9d3ffd-6639-4b2c-9e8f-718948a0980f-provider-credentials\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:25.993613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.993581 5607 generic.go:356] "Generic (PLEG): container finished" podID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerID="e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9" exitCode=0 Oct 30 22:05:25.993754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.993629 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" Oct 30 22:05:25.993754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.993663 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" event={"ID":"3d9d3ffd-6639-4b2c-9e8f-718948a0980f","Type":"ContainerDied","Data":"e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9"} Oct 30 22:05:25.993754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.993715 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th" event={"ID":"3d9d3ffd-6639-4b2c-9e8f-718948a0980f","Type":"ContainerDied","Data":"c498ba0c6be82ad60e03ca2792a36e67293328dc0c31e96dc5e8c23ededaf786"} Oct 30 22:05:25.993754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:25.993737 5607 scope.go:122] "RemoveContainer" containerID="e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9" Oct 30 22:05:26.004501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:26.004487 5607 scope.go:122] "RemoveContainer" containerID="e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9" Oct 30 22:05:26.004732 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:26.004715 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9\": container with ID starting with e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9 not found: ID does not exist" containerID="e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9" Oct 30 22:05:26.004778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:26.004739 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9"} err="failed to get container status \"e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9\": rpc error: code = NotFound desc = could not find container \"e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9\": container with ID starting with e18e42c79b6c7c0696bb0ba1ba1475d91347a4a2386e655800a3419508776fe9 not found: ID does not exist" Oct 30 22:05:26.163254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:26.163234 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th"] Oct 30 22:05:26.213838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:26.213820 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7474dfb69b-gj7th"] Oct 30 22:05:26.603493 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:26.603469 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-594b013fa537: secret "agc-cred-594b013fa537" not found Oct 30 22:05:26.603493 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:26.603492 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-594b013fa537" not found Oct 30 22:05:26.604025 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:26.603550 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:42.603531486 +0000 UTC m=+1122.004087601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-594b013fa537" not found Oct 30 22:05:26.869947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:26.869872 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" path="/var/lib/kubelet/pods/3d9d3ffd-6639-4b2c-9e8f-718948a0980f/volumes" Oct 30 22:05:32.970291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:32.970257 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:39.644683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.644654 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58"] Oct 30 22:05:39.645235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.645214 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" Oct 30 22:05:39.645288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.645236 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" Oct 30 22:05:39.645323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.645318 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9d3ffd-6639-4b2c-9e8f-718948a0980f" containerName="payload-processing" Oct 30 22:05:39.649992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.649976 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.729917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.729890 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.729935 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.729981 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.730129 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqsd\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.730170 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.730260 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.730295 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.730362 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.730426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.730415 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.754065 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.754037 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58"] Oct 30 22:05:39.831142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831113 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.831272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831184 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.831272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831208 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.831272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831229 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqsd\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.831395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831280 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.831395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831335 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.831395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831368 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831428 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831469 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831531 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831580 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831816 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.831820 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.832049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.832023 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.833769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.833747 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.834060 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.834040 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.898275 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.898222 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqsd\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.900568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.900545 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:39.963411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:39.963392 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:40.385174 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:05:40.385140 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6575f77f_d61d_454d_9ed3_4c5fcf68c640.slice/crio-331a0d8a6bf0332ea614777a56a8fca852c56500316a1b84444b86c611531951 WatchSource:0}: Error finding container 331a0d8a6bf0332ea614777a56a8fca852c56500316a1b84444b86c611531951: Status 404 returned error can't find the container with id 331a0d8a6bf0332ea614777a56a8fca852c56500316a1b84444b86c611531951 Oct 30 22:05:40.387561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:40.387529 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:40.387654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:40.387596 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:40.387654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:40.387623 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:40.388577 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:40.388550 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58"] Oct 30 22:05:41.089905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:41.089872 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" event={"ID":"6575f77f-d61d-454d-9ed3-4c5fcf68c640","Type":"ContainerStarted","Data":"59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be"} Oct 30 22:05:41.090366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:41.089920 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" event={"ID":"6575f77f-d61d-454d-9ed3-4c5fcf68c640","Type":"ContainerStarted","Data":"331a0d8a6bf0332ea614777a56a8fca852c56500316a1b84444b86c611531951"} Oct 30 22:05:41.280084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:41.280034 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podStartSLOduration=2.280019775 podStartE2EDuration="2.280019775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:41.277854297 +0000 UTC m=+1120.678410418" watchObservedRunningTime="2026-10-30 22:05:41.280019775 +0000 UTC m=+1120.680575970" Oct 30 22:05:41.964385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:41.964357 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:41.966370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:41.966343 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:05:41.966490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:41.966393 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:05:42.664672 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:42.664645 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-594b013fa537: secret "agc-cred-594b013fa537" not found Oct 30 22:05:42.664672 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:42.664669 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-594b013fa537" not found Oct 30 22:05:42.665135 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:42.664731 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:14.664715785 +0000 UTC m=+1154.065271888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-594b013fa537" not found Oct 30 22:05:42.964172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:42.964142 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:05:42.964344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:42.964191 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:05:42.968336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:42.968310 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:43.638507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.638474 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx"] Oct 30 22:05:43.642721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.642703 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.752260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.752236 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"distinct-2-w4-e524ce-kserve-self-signed-certs\"" Oct 30 22:05:43.755235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.755211 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx"] Oct 30 22:05:43.778568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.778540 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.778688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.778579 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.778688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.778666 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.778796 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.778710 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvzg\" (UniqueName: \"kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.778834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.778794 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.779050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.779035 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880055 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880098 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880115 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880143 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880166 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvzg\" (UniqueName: \"kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880195 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880585 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.880810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.880795 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.881354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.881333 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.883005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.882983 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.883493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.883475 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.946864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.946835 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvzg\" (UniqueName: \"kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg\") pod \"distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.956686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.956661 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:43.964503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.964481 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:05:43.964590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:43.964524 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:05:44.352766 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:05:44.352720 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a6e63ab_61f5_4da2_80e1_eac2983b35fe.slice/crio-972a97287b47316a5ad33653dd425d043a656e06837f53811771099e0d5ae200 WatchSource:0}: Error finding container 972a97287b47316a5ad33653dd425d043a656e06837f53811771099e0d5ae200: Status 404 returned error can't find the container with id 972a97287b47316a5ad33653dd425d043a656e06837f53811771099e0d5ae200 Oct 30 22:05:44.356012 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:44.355980 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx"] Oct 30 22:05:44.964074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:44.964043 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:05:44.964473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:44.964091 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:05:45.147143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:45.147111 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" event={"ID":"6a6e63ab-61f5-4da2-80e1-eac2983b35fe","Type":"ContainerStarted","Data":"86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478"} Oct 30 22:05:45.147307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:45.147154 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" event={"ID":"6a6e63ab-61f5-4da2-80e1-eac2983b35fe","Type":"ContainerStarted","Data":"972a97287b47316a5ad33653dd425d043a656e06837f53811771099e0d5ae200"} Oct 30 22:05:45.282694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:45.282598 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podStartSLOduration=2.282583958 podStartE2EDuration="2.282583958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:45.281538332 +0000 UTC m=+1124.682094449" watchObservedRunningTime="2026-10-30 22:05:45.282583958 +0000 UTC m=+1124.683140079" Oct 30 22:05:45.957015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:45.956981 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:45.964368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:45.964341 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:05:45.964766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:45.964392 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:05:46.964468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:46.964402 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:05:46.964849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:46.964501 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:05:47.968939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:47.968910 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:47.969421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:47.969402 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:47.970060 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:47.970038 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:05:52.863026 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:05:52.862989 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:05:52.968424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:52.968387 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:53.957001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:53.956941 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:53.969311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:53.969288 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:05:54.231716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:05:54.231629 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:06:01.072213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:01.072184 5607 scope.go:122] "RemoveContainer" containerID="f9b0f7ae4ee8db6c7e636d9a523a6265c228027e7f51f2bcc09d69ec8367d1cc" Oct 30 22:06:01.081079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:01.081062 5607 scope.go:122] "RemoveContainer" containerID="687abd9cc7806a2bd6ab6b862819ce36948b2c43385da7c885b98b942223eb6e" Oct 30 22:06:02.968494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:02.968461 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:09.026651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.026619 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5"] Oct 30 22:06:09.032544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.032518 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098255 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098293 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098309 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098387 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098444 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098507 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098523 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jzd\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098561 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.098669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.098630 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.137741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.137719 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5"] Oct 30 22:06:09.199914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.199878 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.199931 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.199992 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200042 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200155 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200219 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200247 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jzd\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200460 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200374 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200460 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200414 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.200878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200829 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.201022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.200942 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.201022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.201011 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.201520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.201493 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.201618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.201515 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.203023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.202998 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.203276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.203258 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.268381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.268357 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jzd\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.268560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.268540 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.345868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.345797 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:09.772791 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:06:09.772763 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00710edf_10cd_4588_8fc4_68fdb78c78e9.slice/crio-f6d494ae92e3ffe60bbdb93015e80d7513a35131ac0a92504b65bb869b5421f1 WatchSource:0}: Error finding container f6d494ae92e3ffe60bbdb93015e80d7513a35131ac0a92504b65bb869b5421f1: Status 404 returned error can't find the container with id f6d494ae92e3ffe60bbdb93015e80d7513a35131ac0a92504b65bb869b5421f1 Oct 30 22:06:09.775551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.775515 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:09.775644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.775585 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:09.775644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.775615 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:09.776446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:09.776427 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5"] Oct 30 22:06:10.316888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:10.316842 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" event={"ID":"00710edf-10cd-4588-8fc4-68fdb78c78e9","Type":"ContainerStarted","Data":"3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a"} Oct 30 22:06:10.317343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:10.317121 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" event={"ID":"00710edf-10cd-4588-8fc4-68fdb78c78e9","Type":"ContainerStarted","Data":"f6d494ae92e3ffe60bbdb93015e80d7513a35131ac0a92504b65bb869b5421f1"} Oct 30 22:06:10.346087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:10.346061 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:10.347797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:10.347774 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:10.347926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:10.347816 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:10.447551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:10.447498 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podStartSLOduration=2.447482473 podStartE2EDuration="2.447482473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:10.444731289 +0000 UTC m=+1149.845287411" watchObservedRunningTime="2026-10-30 22:06:10.447482473 +0000 UTC m=+1149.848038594" Oct 30 22:06:11.346456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:11.346425 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:11.346986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:11.346472 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:12.346156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:12.346123 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:12.346341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:12.346180 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:12.968491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:12.968456 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:13.346532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:13.346453 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:13.346532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:13.346516 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:14.347126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:14.347094 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:14.347583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:14.347154 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:14.672004 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:06:14.671912 5607 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-594b013fa537: secret "agc-cred-594b013fa537" not found Oct 30 22:06:14.672004 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:06:14.671935 5607 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd: secret "agc-cred-594b013fa537" not found Oct 30 22:06:14.672228 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:06:14.672013 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials podName:775b5cdb-1352-49c9-8362-abf8572966a5 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:18.671993674 +0000 UTC m=+1218.072549777 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials") pod "payload-processing-external-model-5ff46798db-xzrpd" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5") : secret "agc-cred-594b013fa537" not found Oct 30 22:06:15.346097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:15.346066 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:15.346274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:15.346130 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:16.346909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:16.346872 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:06:16.347355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:16.346945 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 22:06:17.350649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:17.350621 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:17.351134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:17.351065 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:17.351662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:17.351644 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:06:22.863716 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:06:22.863681 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:06:22.968172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:22.968130 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:23.038142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.038114 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp"] Oct 30 22:06:23.042334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.042315 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.106710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.102823 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:23.106710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.102988 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:06:23.155371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.155338 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp"] Oct 30 22:06:23.166108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.166081 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5879s\" (UniqueName: \"kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.166284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.166231 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.166354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.166322 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.267420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.267347 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5879s\" (UniqueName: \"kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.267547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.267447 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.267547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.267508 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.268087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.268069 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.269998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.269968 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.331484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.331454 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5879s\" (UniqueName: \"kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s\") pod \"maas-api-revoke-sub-7e4bb830-4xfvp\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.356434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.356411 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:23.731690 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:06:23.731656 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd8cbe8_1932_41b2_b43b_a24eee44bf4b.slice/crio-92cda3edca3ad1c19f70059ce8f4816b14846de016dafb4b85f23d2e6cff0f00 WatchSource:0}: Error finding container 92cda3edca3ad1c19f70059ce8f4816b14846de016dafb4b85f23d2e6cff0f00: Status 404 returned error can't find the container with id 92cda3edca3ad1c19f70059ce8f4816b14846de016dafb4b85f23d2e6cff0f00 Oct 30 22:06:23.734976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.734938 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp"] Oct 30 22:06:23.863623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:23.863595 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:24.411684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:24.411578 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" event={"ID":"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b","Type":"ContainerStarted","Data":"79992eacd434db60536d10b0cc5dc0c6d00e8c097a38315118f2e7dc1c2d0afa"} Oct 30 22:06:24.411684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:24.411649 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" event={"ID":"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b","Type":"ContainerStarted","Data":"92cda3edca3ad1c19f70059ce8f4816b14846de016dafb4b85f23d2e6cff0f00"} Oct 30 22:06:24.531291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:24.531238 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" podStartSLOduration=2.531224016 podStartE2EDuration="2.531224016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:24.530232112 +0000 UTC m=+1163.930788233" watchObservedRunningTime="2026-10-30 22:06:24.531224016 +0000 UTC m=+1163.931780206" Oct 30 22:06:25.420385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:25.420352 5607 generic.go:356] "Generic (PLEG): container finished" podID="8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" containerID="79992eacd434db60536d10b0cc5dc0c6d00e8c097a38315118f2e7dc1c2d0afa" exitCode=0 Oct 30 22:06:25.420788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:25.420416 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" event={"ID":"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b","Type":"ContainerDied","Data":"79992eacd434db60536d10b0cc5dc0c6d00e8c097a38315118f2e7dc1c2d0afa"} Oct 30 22:06:26.587884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.587856 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:26.708325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.708299 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca\") pod \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " Oct 30 22:06:26.708493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.708392 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca\") pod \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " Oct 30 22:06:26.708493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.708448 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s\" (UniqueName: \"kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s\") pod \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\" (UID: \"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b\") " Oct 30 22:06:26.708638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.708612 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca" pod "8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" (UID: "8dd8cbe8-1932-41b2-b43b-a24eee44bf4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:26.709111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.709041 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:26.715124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.715095 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s" pod "8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" (UID: "8dd8cbe8-1932-41b2-b43b-a24eee44bf4b"). InnerVolumeSpecName "kube-api-access-5879s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:26.715124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.715106 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca" pod "8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" (UID: "8dd8cbe8-1932-41b2-b43b-a24eee44bf4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:26.810841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.810813 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:26.810841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:26.810839 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5879s\" (UniqueName: \"kubernetes.io/projected/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b-kube-api-access-5879s\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:27.435503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:27.435474 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" Oct 30 22:06:27.435804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:27.435477 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp" event={"ID":"8dd8cbe8-1932-41b2-b43b-a24eee44bf4b","Type":"ContainerDied","Data":"92cda3edca3ad1c19f70059ce8f4816b14846de016dafb4b85f23d2e6cff0f00"} Oct 30 22:06:27.435804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:27.435602 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92cda3edca3ad1c19f70059ce8f4816b14846de016dafb4b85f23d2e6cff0f00" Oct 30 22:06:28.294093 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.294060 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48"] Oct 30 22:06:28.294659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.294645 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" containerName="revoke-keys" Oct 30 22:06:28.294711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.294660 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" containerName="revoke-keys" Oct 30 22:06:28.294764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.294745 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" containerName="revoke-keys" Oct 30 22:06:28.298272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.298245 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.353767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.353743 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:28.406458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.406438 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:06:28.409323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.409302 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48"] Oct 30 22:06:28.429588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.429567 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.429782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.429763 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.429887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.429870 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflxz\" (UniqueName: \"kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.531323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.531301 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.531437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.531342 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflxz\" (UniqueName: \"kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.531437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.531403 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.532064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.532047 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.533784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.533765 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.596395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.596341 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflxz\" (UniqueName: \"kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz\") pod \"maas-api-revoke-sub-f9e521d2-ngr48\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:28.612074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:28.612056 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:29.031160 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:06:29.031134 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6a7c86_7b95_4f9a_8df6_4ccf7b99557f.slice/crio-9b2bff2980278d8244ce00b8e71bdd00894948b12e71b626a5b84150bca830a2 WatchSource:0}: Error finding container 9b2bff2980278d8244ce00b8e71bdd00894948b12e71b626a5b84150bca830a2: Status 404 returned error can't find the container with id 9b2bff2980278d8244ce00b8e71bdd00894948b12e71b626a5b84150bca830a2 Oct 30 22:06:29.032715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:29.032691 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48"] Oct 30 22:06:29.448773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:29.448739 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" event={"ID":"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f","Type":"ContainerStarted","Data":"2508c8aaf9e02f88ba9990f0742f29f7e8163be172dc6390fe03fcaa5a9445c8"} Oct 30 22:06:29.449231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:29.448802 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" event={"ID":"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f","Type":"ContainerStarted","Data":"9b2bff2980278d8244ce00b8e71bdd00894948b12e71b626a5b84150bca830a2"} Oct 30 22:06:29.573430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:29.573356 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" podStartSLOduration=1.5733433529999998 podStartE2EDuration="1.573343353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:29.572384473 +0000 UTC m=+1168.972940601" watchObservedRunningTime="2026-10-30 22:06:29.573343353 +0000 UTC m=+1168.973899527" Oct 30 22:06:30.456976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:30.456920 5607 generic.go:356] "Generic (PLEG): container finished" podID="ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" containerID="2508c8aaf9e02f88ba9990f0742f29f7e8163be172dc6390fe03fcaa5a9445c8" exitCode=0 Oct 30 22:06:30.457376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:30.456984 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" event={"ID":"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f","Type":"ContainerDied","Data":"2508c8aaf9e02f88ba9990f0742f29f7e8163be172dc6390fe03fcaa5a9445c8"} Oct 30 22:06:31.611107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.611078 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:31.762733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.762649 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca\") pod \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " Oct 30 22:06:31.762733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.762715 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz\" (UniqueName: \"kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz\") pod \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " Oct 30 22:06:31.763000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.762783 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca\") pod \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\" (UID: \"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f\") " Oct 30 22:06:31.763211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.763186 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca" pod "ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" (UID: "ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:31.763383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.763363 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:31.765276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.765247 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca" pod "ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" (UID: "ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:31.765450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.765428 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz" pod "ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" (UID: "ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f"). InnerVolumeSpecName "kube-api-access-lflxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:31.864211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.864182 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:31.864211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:31.864207 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflxz\" (UniqueName: \"kubernetes.io/projected/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f-kube-api-access-lflxz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:32.471373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:32.471351 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" Oct 30 22:06:32.471542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:32.471344 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48" event={"ID":"ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f","Type":"ContainerDied","Data":"9b2bff2980278d8244ce00b8e71bdd00894948b12e71b626a5b84150bca830a2"} Oct 30 22:06:32.471542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:32.471478 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b2bff2980278d8244ce00b8e71bdd00894948b12e71b626a5b84150bca830a2" Oct 30 22:06:33.863769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:33.863730 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:34.372278 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:34.372244 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd"] Oct 30 22:06:39.201140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.201107 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79"] Oct 30 22:06:39.201720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.201704 5607 cpu_manager.go:414] "containerMap: removing container" podUID="ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" containerName="revoke-keys" Oct 30 22:06:39.201767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.201722 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" containerName="revoke-keys" Oct 30 22:06:39.201851 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.201842 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" containerName="revoke-keys" Oct 30 22:06:39.205634 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.205617 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.267990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.264937 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79"] Oct 30 22:06:39.295663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.295637 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.295803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.295700 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6mr\" (UniqueName: \"kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.295879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.295850 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.311652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.311629 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:39.311652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.311630 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:06:39.396814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.396795 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.396926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.396878 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.396926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.396908 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6mr\" (UniqueName: \"kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.397495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.397476 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.399383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.399365 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.459541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.459493 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6mr\" (UniqueName: \"kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr\") pod \"maas-api-revoke-sub-5824372f-nnt79\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.518860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.518841 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:39.893706 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:06:39.893665 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393f38aa_e2f3_4771_9282_f3273194438c.slice/crio-16a7dd811f1e6b88543411b86d0b7598626ae0d1e13c0ab5d92f31dbf59e34ee WatchSource:0}: Error finding container 16a7dd811f1e6b88543411b86d0b7598626ae0d1e13c0ab5d92f31dbf59e34ee: Status 404 returned error can't find the container with id 16a7dd811f1e6b88543411b86d0b7598626ae0d1e13c0ab5d92f31dbf59e34ee Oct 30 22:06:39.898143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:39.898120 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79"] Oct 30 22:06:40.560205 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:40.560120 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" event={"ID":"393f38aa-e2f3-4771-9282-f3273194438c","Type":"ContainerStarted","Data":"6e38b6df3bb0962bb9b1efd65d8d36bab590a6d289c91e8def2ca0f4b3a93b38"} Oct 30 22:06:40.560205 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:40.560185 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" event={"ID":"393f38aa-e2f3-4771-9282-f3273194438c","Type":"ContainerStarted","Data":"16a7dd811f1e6b88543411b86d0b7598626ae0d1e13c0ab5d92f31dbf59e34ee"} Oct 30 22:06:40.713671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:40.713575 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" podStartSLOduration=1.713560499 podStartE2EDuration="1.713560499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:40.712438299 +0000 UTC m=+1180.112994420" watchObservedRunningTime="2026-10-30 22:06:40.713560499 +0000 UTC m=+1180.114116620" Oct 30 22:06:41.594257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:41.594010 5607 generic.go:356] "Generic (PLEG): container finished" podID="393f38aa-e2f3-4771-9282-f3273194438c" containerID="6e38b6df3bb0962bb9b1efd65d8d36bab590a6d289c91e8def2ca0f4b3a93b38" exitCode=0 Oct 30 22:06:41.594257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:41.594184 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" event={"ID":"393f38aa-e2f3-4771-9282-f3273194438c","Type":"ContainerDied","Data":"6e38b6df3bb0962bb9b1efd65d8d36bab590a6d289c91e8def2ca0f4b3a93b38"} Oct 30 22:06:42.770227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.770200 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:42.839254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.839227 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr\" (UniqueName: \"kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr\") pod \"393f38aa-e2f3-4771-9282-f3273194438c\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " Oct 30 22:06:42.839385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.839329 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca\") pod \"393f38aa-e2f3-4771-9282-f3273194438c\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " Oct 30 22:06:42.839443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.839431 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca\") pod \"393f38aa-e2f3-4771-9282-f3273194438c\" (UID: \"393f38aa-e2f3-4771-9282-f3273194438c\") " Oct 30 22:06:42.839724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.839689 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca" pod "393f38aa-e2f3-4771-9282-f3273194438c" (UID: "393f38aa-e2f3-4771-9282-f3273194438c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:42.839942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.839926 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:42.841390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.841360 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr" pod "393f38aa-e2f3-4771-9282-f3273194438c" (UID: "393f38aa-e2f3-4771-9282-f3273194438c"). InnerVolumeSpecName "kube-api-access-mk6mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:42.841683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.841662 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca" pod "393f38aa-e2f3-4771-9282-f3273194438c" (UID: "393f38aa-e2f3-4771-9282-f3273194438c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:42.941552 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.941525 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk6mr\" (UniqueName: \"kubernetes.io/projected/393f38aa-e2f3-4771-9282-f3273194438c-kube-api-access-mk6mr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:42.941552 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:42.941549 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393f38aa-e2f3-4771-9282-f3273194438c-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:43.609385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:43.609353 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" Oct 30 22:06:43.609574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:43.609352 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79" event={"ID":"393f38aa-e2f3-4771-9282-f3273194438c","Type":"ContainerDied","Data":"16a7dd811f1e6b88543411b86d0b7598626ae0d1e13c0ab5d92f31dbf59e34ee"} Oct 30 22:06:43.609574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:43.609488 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16a7dd811f1e6b88543411b86d0b7598626ae0d1e13c0ab5d92f31dbf59e34ee" Oct 30 22:06:43.864208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:43.864133 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:45.930538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.930504 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-9ld8d"] Oct 30 22:06:45.931737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.931711 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:06:45.964642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.964615 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59fbf69455-5km7v"] Oct 30 22:06:45.965187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.965173 5607 cpu_manager.go:414] "containerMap: removing container" podUID="393f38aa-e2f3-4771-9282-f3273194438c" containerName="revoke-keys" Oct 30 22:06:45.965252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.965189 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393f38aa-e2f3-4771-9282-f3273194438c" containerName="revoke-keys" Oct 30 22:06:45.965335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.965326 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="393f38aa-e2f3-4771-9282-f3273194438c" containerName="revoke-keys" Oct 30 22:06:45.969097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:45.969082 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.022977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.022926 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a90e87ee4cc5\"" Oct 30 22:06:46.022977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.022928 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7bac358c545e\"" Oct 30 22:06:46.047405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.047377 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59fbf69455-5km7v"] Oct 30 22:06:46.074931 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.074902 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.075108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.075082 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.075170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.075123 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.075170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.075160 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slth6\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.176192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.176158 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.176329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.176252 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.176329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.176282 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.176329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.176314 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slth6\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.176911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.176886 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.177067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.176942 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.179094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.179069 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.238847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.238820 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slth6\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6\") pod \"payload-processing-external-model-59fbf69455-5km7v\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.283691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.283667 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:46.653701 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:06:46.653674 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3c86cf_55b9_4ecc_ab77_654a56fb1a51.slice/crio-df07660bb5c250b6153730690ebb5189139bfa1af86692eb2a64073529e74a15 WatchSource:0}: Error finding container df07660bb5c250b6153730690ebb5189139bfa1af86692eb2a64073529e74a15: Status 404 returned error can't find the container with id df07660bb5c250b6153730690ebb5189139bfa1af86692eb2a64073529e74a15 Oct 30 22:06:46.658576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:46.658553 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59fbf69455-5km7v"] Oct 30 22:06:47.635712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:47.635670 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" event={"ID":"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51","Type":"ContainerStarted","Data":"12cb02e039f9b6cbab66d0a81749500b98da06eed689e0b4db1e85ab0ea2dffd"} Oct 30 22:06:47.636151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:47.635741 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:47.636151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:47.635760 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" event={"ID":"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51","Type":"ContainerStarted","Data":"df07660bb5c250b6153730690ebb5189139bfa1af86692eb2a64073529e74a15"} Oct 30 22:06:47.771618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:47.767262 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" podStartSLOduration=2.767244794 podStartE2EDuration="2.767244794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.761130766 +0000 UTC m=+1187.161686886" watchObservedRunningTime="2026-10-30 22:06:47.767244794 +0000 UTC m=+1187.167800917" Oct 30 22:06:49.649117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:49.649073 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerName="main" containerID="cri-o://def6a689b759b30beff538e6d069994f61e4965c3a83ee7b2db5f65463a44f55" gracePeriod=45 Oct 30 22:06:50.274027 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.273049 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:50.481978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.481078 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:50.495983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.495041 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:50.495983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.495446 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": dial tcp 10.133.0.80:8000: connect: connection refused" Oct 30 22:06:50.495983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.495554 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:06:50.703974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.701700 5607 generic.go:356] "Generic (PLEG): container finished" podID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerID="def6a689b759b30beff538e6d069994f61e4965c3a83ee7b2db5f65463a44f55" exitCode=0 Oct 30 22:06:50.703974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.701902 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" event={"ID":"535e6f8e-5e47-461c-b469-ed0ae0bf6388","Type":"ContainerDied","Data":"def6a689b759b30beff538e6d069994f61e4965c3a83ee7b2db5f65463a44f55"} Oct 30 22:06:50.960976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.957556 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" containerID="cri-o://e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40" gracePeriod=25 Oct 30 22:06:50.993577 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:50.992455 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.176413 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home\") pod \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.176484 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir\") pod \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.176557 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs\" (UniqueName: \"kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs\") pod \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.176627 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache\") pod \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.176785 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg\" (UniqueName: \"kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg\") pod \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.176831 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm\") pod \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\" (UID: \"535e6f8e-5e47-461c-b469-ed0ae0bf6388\") " Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.178554 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache" pod "535e6f8e-5e47-461c-b469-ed0ae0bf6388" (UID: "535e6f8e-5e47-461c-b469-ed0ae0bf6388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.178805 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir" pod "535e6f8e-5e47-461c-b469-ed0ae0bf6388" (UID: "535e6f8e-5e47-461c-b469-ed0ae0bf6388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:51.181018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.178872 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home" pod "535e6f8e-5e47-461c-b469-ed0ae0bf6388" (UID: "535e6f8e-5e47-461c-b469-ed0ae0bf6388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:51.184194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.184144 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs" pod "535e6f8e-5e47-461c-b469-ed0ae0bf6388" (UID: "535e6f8e-5e47-461c-b469-ed0ae0bf6388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:51.184436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.184411 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm" pod "535e6f8e-5e47-461c-b469-ed0ae0bf6388" (UID: "535e6f8e-5e47-461c-b469-ed0ae0bf6388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:51.185418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.185391 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg" pod "535e6f8e-5e47-461c-b469-ed0ae0bf6388" (UID: "535e6f8e-5e47-461c-b469-ed0ae0bf6388"). InnerVolumeSpecName "kube-api-access-nsfwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:51.283980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.283831 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsfwg\" (UniqueName: \"kubernetes.io/projected/535e6f8e-5e47-461c-b469-ed0ae0bf6388-kube-api-access-nsfwg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.283980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.283868 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.283980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.283884 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.283980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.283897 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.283980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.283913 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e6f8e-5e47-461c-b469-ed0ae0bf6388-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.283980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.283927 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e6f8e-5e47-461c-b469-ed0ae0bf6388-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.641924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.641627 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690034 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials\") pod \"359ab5c8-014e-421c-b13d-ce3fbcb37506\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690129 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume\") pod \"359ab5c8-014e-421c-b13d-ce3fbcb37506\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690211 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k\") pod \"359ab5c8-014e-421c-b13d-ce3fbcb37506\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690281 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume\") pod \"359ab5c8-014e-421c-b13d-ce3fbcb37506\" (UID: \"359ab5c8-014e-421c-b13d-ce3fbcb37506\") " Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690477 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume" pod "359ab5c8-014e-421c-b13d-ce3fbcb37506" (UID: "359ab5c8-014e-421c-b13d-ce3fbcb37506"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690752 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.691975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.690877 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume" pod "359ab5c8-014e-421c-b13d-ce3fbcb37506" (UID: "359ab5c8-014e-421c-b13d-ce3fbcb37506"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:51.692686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.692664 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k" pod "359ab5c8-014e-421c-b13d-ce3fbcb37506" (UID: "359ab5c8-014e-421c-b13d-ce3fbcb37506"). InnerVolumeSpecName "kube-api-access-9wg5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:51.692820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.692778 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials" pod "359ab5c8-014e-421c-b13d-ce3fbcb37506" (UID: "359ab5c8-014e-421c-b13d-ce3fbcb37506"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:51.751976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.742879 5607 generic.go:356] "Generic (PLEG): container finished" podID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerID="e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40" exitCode=0 Oct 30 22:06:51.761981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.753298 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" Oct 30 22:06:51.763090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.762158 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" event={"ID":"359ab5c8-014e-421c-b13d-ce3fbcb37506","Type":"ContainerDied","Data":"e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40"} Oct 30 22:06:51.763090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.762230 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-9ld8d" event={"ID":"359ab5c8-014e-421c-b13d-ce3fbcb37506","Type":"ContainerDied","Data":"a452637c594bfc19c7c3632ee58ceef57fdc5caac6e97926a9061c1147f921c9"} Oct 30 22:06:51.763090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.762263 5607 scope.go:122] "RemoveContainer" containerID="e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40" Oct 30 22:06:51.785970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.785267 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" event={"ID":"535e6f8e-5e47-461c-b469-ed0ae0bf6388","Type":"ContainerDied","Data":"45a2366b651bdd9985a52eca0b4f13a8d4d5b84315ddcc79388532a8fed54be8"} Oct 30 22:06:51.785970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.785358 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd" Oct 30 22:06:51.791865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.791839 5607 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-provider-credentials\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.791998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.791870 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wg5k\" (UniqueName: \"kubernetes.io/projected/359ab5c8-014e-421c-b13d-ce3fbcb37506-kube-api-access-9wg5k\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.791998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.791887 5607 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/359ab5c8-014e-421c-b13d-ce3fbcb37506-routing-overlay-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:51.794429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.794410 5607 scope.go:122] "RemoveContainer" containerID="e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40" Oct 30 22:06:51.794796 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:06:51.794770 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40\": container with ID starting with e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40 not found: ID does not exist" containerID="e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40" Oct 30 22:06:51.794853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.794807 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40"} err="failed to get container status \"e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40\": rpc error: code = NotFound desc = could not find container \"e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40\": container with ID starting with e440a317d9cec73e7cc7abaaa37d195443209118ffbbf658864985e8b1b3fa40 not found: ID does not exist" Oct 30 22:06:51.794853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:51.794833 5607 scope.go:122] "RemoveContainer" containerID="def6a689b759b30beff538e6d069994f61e4965c3a83ee7b2db5f65463a44f55" Oct 30 22:06:52.032992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:52.031010 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-9ld8d"] Oct 30 22:06:52.065986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:52.065015 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-9ld8d"] Oct 30 22:06:52.162032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:52.158897 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd"] Oct 30 22:06:52.165283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:52.165082 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-a-d3c7fa24-kserve-778868f7d7-lzshd"] Oct 30 22:06:52.863342 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:06:52.863301 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:06:52.870040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:52.870017 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" path="/var/lib/kubelet/pods/359ab5c8-014e-421c-b13d-ce3fbcb37506/volumes" Oct 30 22:06:52.870430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:52.870411 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" path="/var/lib/kubelet/pods/535e6f8e-5e47-461c-b469-ed0ae0bf6388/volumes" Oct 30 22:06:56.340551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.340520 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl"] Oct 30 22:06:56.341079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.341065 5607 cpu_manager.go:414] "containerMap: removing container" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerName="main" Oct 30 22:06:56.341171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.341080 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerName="main" Oct 30 22:06:56.341171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.341095 5607 cpu_manager.go:414] "containerMap: removing container" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" Oct 30 22:06:56.341171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.341100 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" Oct 30 22:06:56.341265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.341200 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="535e6f8e-5e47-461c-b469-ed0ae0bf6388" containerName="main" Oct 30 22:06:56.341265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.341207 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="359ab5c8-014e-421c-b13d-ce3fbcb37506" containerName="payload-processing" Oct 30 22:06:56.346214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.346198 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.407335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.407310 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl"] Oct 30 22:06:56.453417 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.453398 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:06:56.453523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.453431 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:06:56.457570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.457539 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.457773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.457723 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l2p\" (UniqueName: \"kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.457836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.457796 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.559514 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.559489 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l2p\" (UniqueName: \"kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.559652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.559538 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.559652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.559606 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.560268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.560207 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.561821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.561800 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.623012 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.622945 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l2p\" (UniqueName: \"kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p\") pod \"maas-api-revoke-sub-0a572ead-pbchl\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:56.659626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:56.659608 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:06:57.032707 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:06:57.032674 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4f1291_fc55_4d1c_a82e_758ca12a19d4.slice/crio-d00ed0d7b47cafe94d9fc6edce39fe5ca67332500c43528a75afc371276ce133 WatchSource:0}: Error finding container d00ed0d7b47cafe94d9fc6edce39fe5ca67332500c43528a75afc371276ce133: Status 404 returned error can't find the container with id d00ed0d7b47cafe94d9fc6edce39fe5ca67332500c43528a75afc371276ce133 Oct 30 22:06:57.036709 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:57.036686 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl"] Oct 30 22:06:57.827378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:57.827343 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" event={"ID":"4d4f1291-fc55-4d1c-a82e-758ca12a19d4","Type":"ContainerStarted","Data":"135fcddfbc9ecaa17a0fb5e898806ddda177657c797aa5cc84dafc32ae2d82fd"} Oct 30 22:06:57.827739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:57.827395 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" event={"ID":"4d4f1291-fc55-4d1c-a82e-758ca12a19d4","Type":"ContainerStarted","Data":"d00ed0d7b47cafe94d9fc6edce39fe5ca67332500c43528a75afc371276ce133"} Oct 30 22:06:57.948525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:57.948460 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" podStartSLOduration=1.948445615 podStartE2EDuration="1.948445615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:57.94540127 +0000 UTC m=+1197.345957401" watchObservedRunningTime="2026-10-30 22:06:57.948445615 +0000 UTC m=+1197.349001738" Oct 30 22:06:58.645623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:58.645599 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:06:58.809007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:58.806173 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd"] Oct 30 22:06:58.836458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:58.836423 5607 generic.go:356] "Generic (PLEG): container finished" podID="4d4f1291-fc55-4d1c-a82e-758ca12a19d4" containerID="135fcddfbc9ecaa17a0fb5e898806ddda177657c797aa5cc84dafc32ae2d82fd" exitCode=0 Oct 30 22:06:58.836846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:58.836547 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" event={"ID":"4d4f1291-fc55-4d1c-a82e-758ca12a19d4","Type":"ContainerDied","Data":"135fcddfbc9ecaa17a0fb5e898806ddda177657c797aa5cc84dafc32ae2d82fd"} Oct 30 22:06:59.991904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:06:59.991882 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:07:00.095697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.095626 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca\") pod \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " Oct 30 22:07:00.095818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.095782 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p\" (UniqueName: \"kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p\") pod \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " Oct 30 22:07:00.095907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.095893 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca\") pod \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\" (UID: \"4d4f1291-fc55-4d1c-a82e-758ca12a19d4\") " Oct 30 22:07:00.096254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.096229 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca" pod "4d4f1291-fc55-4d1c-a82e-758ca12a19d4" (UID: "4d4f1291-fc55-4d1c-a82e-758ca12a19d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:00.097778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.097753 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca" pod "4d4f1291-fc55-4d1c-a82e-758ca12a19d4" (UID: "4d4f1291-fc55-4d1c-a82e-758ca12a19d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:00.097881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.097862 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p" pod "4d4f1291-fc55-4d1c-a82e-758ca12a19d4" (UID: "4d4f1291-fc55-4d1c-a82e-758ca12a19d4"). InnerVolumeSpecName "kube-api-access-p4l2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:00.197562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.197533 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:00.197562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.197560 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:00.197714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.197569 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4l2p\" (UniqueName: \"kubernetes.io/projected/4d4f1291-fc55-4d1c-a82e-758ca12a19d4-kube-api-access-p4l2p\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:00.850607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.850583 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" Oct 30 22:07:00.850767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.850585 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl" event={"ID":"4d4f1291-fc55-4d1c-a82e-758ca12a19d4","Type":"ContainerDied","Data":"d00ed0d7b47cafe94d9fc6edce39fe5ca67332500c43528a75afc371276ce133"} Oct 30 22:07:00.850767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:00.850722 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d00ed0d7b47cafe94d9fc6edce39fe5ca67332500c43528a75afc371276ce133" Oct 30 22:07:01.014820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:01.014793 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:07:01.023288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:01.023271 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:07:03.807317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:03.807263 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" podUID="775b5cdb-1352-49c9-8362-abf8572966a5" containerName="payload-processing" containerID="cri-o://e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70" gracePeriod=25 Oct 30 22:07:04.068776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.068722 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:07:04.136772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.136746 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g\") pod \"775b5cdb-1352-49c9-8362-abf8572966a5\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " Oct 30 22:07:04.136942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.136923 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume\") pod \"775b5cdb-1352-49c9-8362-abf8572966a5\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " Oct 30 22:07:04.137028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.136991 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials\") pod \"775b5cdb-1352-49c9-8362-abf8572966a5\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " Oct 30 22:07:04.137068 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.137041 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume\") pod \"775b5cdb-1352-49c9-8362-abf8572966a5\" (UID: \"775b5cdb-1352-49c9-8362-abf8572966a5\") " Oct 30 22:07:04.137395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.137361 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume" pod "775b5cdb-1352-49c9-8362-abf8572966a5" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:04.137653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.137477 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume" pod "775b5cdb-1352-49c9-8362-abf8572966a5" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:04.137653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.137566 5607 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-routing-overlay-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.137653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.137585 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775b5cdb-1352-49c9-8362-abf8572966a5-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.139106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.139080 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g" pod "775b5cdb-1352-49c9-8362-abf8572966a5" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5"). InnerVolumeSpecName "kube-api-access-bkb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:04.139262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.139244 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials" pod "775b5cdb-1352-49c9-8362-abf8572966a5" (UID: "775b5cdb-1352-49c9-8362-abf8572966a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:04.238809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.238782 5607 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-provider-credentials\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.238809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.238806 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkb5g\" (UniqueName: \"kubernetes.io/projected/775b5cdb-1352-49c9-8362-abf8572966a5-kube-api-access-bkb5g\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:04.876178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.876145 5607 generic.go:356] "Generic (PLEG): container finished" podID="775b5cdb-1352-49c9-8362-abf8572966a5" containerID="e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70" exitCode=0 Oct 30 22:07:04.876648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.876203 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" Oct 30 22:07:04.876648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.876244 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" event={"ID":"775b5cdb-1352-49c9-8362-abf8572966a5","Type":"ContainerDied","Data":"e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70"} Oct 30 22:07:04.876648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.876307 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd" event={"ID":"775b5cdb-1352-49c9-8362-abf8572966a5","Type":"ContainerDied","Data":"de405b9600fa697d978f0725f71af3d6af18ef6e238f3bce1bb140dacc9710e4"} Oct 30 22:07:04.876648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.876337 5607 scope.go:122] "RemoveContainer" containerID="e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70" Oct 30 22:07:04.886156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.886137 5607 scope.go:122] "RemoveContainer" containerID="e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70" Oct 30 22:07:04.886436 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:07:04.886414 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70\": container with ID starting with e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70 not found: ID does not exist" containerID="e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70" Oct 30 22:07:04.886516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:04.886447 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70"} err="failed to get container status \"e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70\": rpc error: code = NotFound desc = could not find container \"e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70\": container with ID starting with e6b5b881d8dc80edbe60e915f3b149152b8a2c81e874341233bf6d0b0b8bfc70 not found: ID does not exist" Oct 30 22:07:05.046510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:05.046477 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd"] Oct 30 22:07:05.050590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:05.050562 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff46798db-xzrpd"] Oct 30 22:07:06.869429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:06.869398 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775b5cdb-1352-49c9-8362-abf8572966a5" path="/var/lib/kubelet/pods/775b5cdb-1352-49c9-8362-abf8572966a5/volumes" Oct 30 22:07:09.074616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.074578 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n"] Oct 30 22:07:09.075214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.075199 5607 cpu_manager.go:414] "containerMap: removing container" podUID="775b5cdb-1352-49c9-8362-abf8572966a5" containerName="payload-processing" Oct 30 22:07:09.075256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.075217 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775b5cdb-1352-49c9-8362-abf8572966a5" containerName="payload-processing" Oct 30 22:07:09.075295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.075258 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4d4f1291-fc55-4d1c-a82e-758ca12a19d4" containerName="revoke-keys" Oct 30 22:07:09.075295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.075263 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4f1291-fc55-4d1c-a82e-758ca12a19d4" containerName="revoke-keys" Oct 30 22:07:09.075361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.075342 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="775b5cdb-1352-49c9-8362-abf8572966a5" containerName="payload-processing" Oct 30 22:07:09.075361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.075356 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4f1291-fc55-4d1c-a82e-758ca12a19d4" containerName="revoke-keys" Oct 30 22:07:09.080358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.080335 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.134316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.134296 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e65019-openshift-default-dockercfg-jjq2k\"" Oct 30 22:07:09.189126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.189101 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n"] Oct 30 22:07:09.191679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191650 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.191824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191698 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.191824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191750 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.191824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191806 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.192054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191839 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.192054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191867 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.192054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191893 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.192054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.191918 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccnjf\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.192054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.192007 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303549 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303584 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303624 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303658 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303684 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303702 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303718 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303738 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccnjf\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.303800 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.304607 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.305210 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.305446 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.305666 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.313976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.306277 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.314849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.314087 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.321590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.321564 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.367688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.367602 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.377834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.377800 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccnjf\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.395371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.395344 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:09.896046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.896009 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n"] Oct 30 22:07:09.916062 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:09.916035 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6669dd_0cbe_4e86_9981_7774b973ba8c.slice/crio-02d62f11b9a4ba9246d392fcfd028b1def0730ab808573eeb6481fad119ed4fb WatchSource:0}: Error finding container 02d62f11b9a4ba9246d392fcfd028b1def0730ab808573eeb6481fad119ed4fb: Status 404 returned error can't find the container with id 02d62f11b9a4ba9246d392fcfd028b1def0730ab808573eeb6481fad119ed4fb Oct 30 22:07:09.919620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.919579 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:09.920270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.920243 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:09.920437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:09.920418 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:10.392652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.392601 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk"] Oct 30 22:07:10.417023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.414262 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.464989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.450285 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk"] Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781059 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781105 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781159 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781205 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781266 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781306 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781335 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781415 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfcn\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.820014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.781445 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.883837 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.883888 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.883940 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884008 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884062 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884102 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884130 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884214 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfcn\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884242 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.884730 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.886144 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.886371 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.886600 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.898979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.887426 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.932296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.916686 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:10.932296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.932197 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" event={"ID":"7c6669dd-0cbe-4e86-9981-7774b973ba8c","Type":"ContainerStarted","Data":"191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408"} Oct 30 22:07:10.932296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.932265 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" event={"ID":"7c6669dd-0cbe-4e86-9981-7774b973ba8c","Type":"ContainerStarted","Data":"02d62f11b9a4ba9246d392fcfd028b1def0730ab808573eeb6481fad119ed4fb"} Oct 30 22:07:10.988978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.987821 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:11.007980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:10.998491 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:11.059368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:11.036992 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfcn\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:11.079442 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:11.079148 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:11.196229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:11.170212 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podStartSLOduration=2.170193346 podStartE2EDuration="2.170193346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:11.161332764 +0000 UTC m=+1210.561888910" watchObservedRunningTime="2026-10-30 22:07:11.170193346 +0000 UTC m=+1210.570749467" Oct 30 22:07:11.418178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:11.418078 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:11.419493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:11.419404 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:11.419847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:11.419817 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:12.135977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.132321 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk"] Oct 30 22:07:12.177975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.166129 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:12.178343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.178311 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:12.178510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.178492 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:12.396773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.396684 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:12.397128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.397005 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:12.917099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.917048 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:12.942085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:12.942042 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 22:07:13.001137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:13.000062 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:13.295438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:13.295401 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" event={"ID":"d77a4b8e-ffef-45e6-84f1-899fb85decad","Type":"ContainerStarted","Data":"19c85dd55a0a8efcb2960482ebd409e4b8e000573c97f08c27a36667611c8892"} Oct 30 22:07:13.295697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:13.295674 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" event={"ID":"d77a4b8e-ffef-45e6-84f1-899fb85decad","Type":"ContainerStarted","Data":"383f9f52c4eac41ffd5f67c169cb2fec0f05c198c08936a8fbea16cf4e1355ed"} Oct 30 22:07:13.399309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:13.399274 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:13.399580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:13.399553 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:14.080559 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:14.080502 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:14.082256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:14.082230 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:07:14.082399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:14.082281 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 22:07:14.435040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:14.417833 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:14.435040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:14.417897 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:15.080326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.080284 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:07:15.080534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.080334 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 22:07:15.418033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.407318 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:15.418033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.407376 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:15.483786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.483735 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podStartSLOduration=5.483721377 podStartE2EDuration="5.483721377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:13.429822794 +0000 UTC m=+1212.830378916" watchObservedRunningTime="2026-10-30 22:07:15.483721377 +0000 UTC m=+1214.884277498" Oct 30 22:07:15.484131 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.484116 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5"] Oct 30 22:07:15.488418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.488400 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.561187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.561120 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5"] Oct 30 22:07:15.581307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.581280 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"distinct-2-w0-d09fff-kserve-self-signed-certs\"" Oct 30 22:07:15.661508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.661468 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.661684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.661592 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plndv\" (UniqueName: \"kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.661684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.661633 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.661765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.661727 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.661837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.661820 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.661887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.661852 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.762779 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.762740 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.762779 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.762783 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.762901 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.763010 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plndv\" (UniqueName: \"kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.763044 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.763090 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.763147 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.763456 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.763655 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.763633 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.765886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.765867 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.766076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.766060 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:15.846742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:15.846677 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plndv\" (UniqueName: \"kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv\") pod \"distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:16.079970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:16.079853 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:07:16.079970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:16.079904 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 22:07:16.137190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:16.137155 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:16.420026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:16.419334 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:16.420026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:16.419398 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:16.556611 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:16.556573 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5591863e_ed98_405d_b7ce_7c0c9eeb099f.slice/crio-9dbd9563f29e1fc53d4944f6e1d14c7febe3263bd9bb22a7309cf14bb1727792 WatchSource:0}: Error finding container 9dbd9563f29e1fc53d4944f6e1d14c7febe3263bd9bb22a7309cf14bb1727792: Status 404 returned error can't find the container with id 9dbd9563f29e1fc53d4944f6e1d14c7febe3263bd9bb22a7309cf14bb1727792 Oct 30 22:07:16.557333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:16.557311 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5"] Oct 30 22:07:17.080892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:17.080853 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:07:17.081110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:17.080916 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 22:07:17.374500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:17.374413 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" event={"ID":"5591863e-ed98-405d-b7ce-7c0c9eeb099f","Type":"ContainerStarted","Data":"156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e"} Oct 30 22:07:17.374500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:17.374456 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" event={"ID":"5591863e-ed98-405d-b7ce-7c0c9eeb099f","Type":"ContainerStarted","Data":"9dbd9563f29e1fc53d4944f6e1d14c7febe3263bd9bb22a7309cf14bb1727792"} Oct 30 22:07:17.410305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:17.410271 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:17.410567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:17.410539 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:18.080704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:18.080672 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:07:18.081096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:18.080727 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 22:07:18.138032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:18.138004 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:18.404156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:18.404072 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:07:18.404407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:18.404379 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 22:07:19.080807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.080769 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:07:19.081353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.080824 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 22:07:19.400188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.400110 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:19.400514 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.400493 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:19.401300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.401278 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:07:19.578109 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.578056 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podStartSLOduration=4.578037001 podStartE2EDuration="4.578037001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:17.517515664 +0000 UTC m=+1216.918071785" watchObservedRunningTime="2026-10-30 22:07:19.578037001 +0000 UTC m=+1218.978593118" Oct 30 22:07:19.940984 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.940929 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz"] Oct 30 22:07:19.946173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:19.946147 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.002671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.002638 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:20.002671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.002642 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:07:20.006991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.006945 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz"] Oct 30 22:07:20.081052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.081026 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmvc\" (UniqueName: \"kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.081482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.081159 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.081482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.081331 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.084812 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.084786 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:20.085253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.085235 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:20.086063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.086047 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:07:20.182162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.182132 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.182263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.182213 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmvc\" (UniqueName: \"kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.182311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.182293 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.182816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.182792 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.185772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.184967 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.248304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.248246 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmvc\" (UniqueName: \"kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc\") pod \"maas-api-revoke-sub-25be4af4-hksfz\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.264793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.264770 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:20.634889 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:20.634859 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19ce1e1_ff8f_4c92_bcce_404325e5f45a.slice/crio-a3adf764a72895d72482ab216f38e8518468d05b175761b77d0b6c19a68f37c9 WatchSource:0}: Error finding container a3adf764a72895d72482ab216f38e8518468d05b175761b77d0b6c19a68f37c9: Status 404 returned error can't find the container with id a3adf764a72895d72482ab216f38e8518468d05b175761b77d0b6c19a68f37c9 Oct 30 22:07:20.638708 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:20.638687 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz"] Oct 30 22:07:21.407889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:21.407855 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" event={"ID":"f19ce1e1-ff8f-4c92-bcce-404325e5f45a","Type":"ContainerStarted","Data":"75d3affd72c423e0aab304d91af53d2fb57836f289c4de4ea05ec80bbf44d7ee"} Oct 30 22:07:21.408405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:21.407917 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" event={"ID":"f19ce1e1-ff8f-4c92-bcce-404325e5f45a","Type":"ContainerStarted","Data":"a3adf764a72895d72482ab216f38e8518468d05b175761b77d0b6c19a68f37c9"} Oct 30 22:07:21.530173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:21.530121 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" podStartSLOduration=2.530106665 podStartE2EDuration="2.530106665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:21.527216611 +0000 UTC m=+1220.927772733" watchObservedRunningTime="2026-10-30 22:07:21.530106665 +0000 UTC m=+1220.930662825" Oct 30 22:07:22.416138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:22.416107 5607 generic.go:356] "Generic (PLEG): container finished" podID="f19ce1e1-ff8f-4c92-bcce-404325e5f45a" containerID="75d3affd72c423e0aab304d91af53d2fb57836f289c4de4ea05ec80bbf44d7ee" exitCode=0 Oct 30 22:07:22.416518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:22.416163 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" event={"ID":"f19ce1e1-ff8f-4c92-bcce-404325e5f45a","Type":"ContainerDied","Data":"75d3affd72c423e0aab304d91af53d2fb57836f289c4de4ea05ec80bbf44d7ee"} Oct 30 22:07:22.862679 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:07:22.862643 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:07:23.564406 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.564385 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:23.618614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.618590 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca\") pod \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " Oct 30 22:07:23.618759 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.618742 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc\" (UniqueName: \"kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc\") pod \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " Oct 30 22:07:23.618926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.618910 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca\") pod \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\" (UID: \"f19ce1e1-ff8f-4c92-bcce-404325e5f45a\") " Oct 30 22:07:23.619243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.619222 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca" pod "f19ce1e1-ff8f-4c92-bcce-404325e5f45a" (UID: "f19ce1e1-ff8f-4c92-bcce-404325e5f45a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:23.619409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.619394 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:23.620809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.620784 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca" pod "f19ce1e1-ff8f-4c92-bcce-404325e5f45a" (UID: "f19ce1e1-ff8f-4c92-bcce-404325e5f45a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:23.620862 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.620792 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc" pod "f19ce1e1-ff8f-4c92-bcce-404325e5f45a" (UID: "f19ce1e1-ff8f-4c92-bcce-404325e5f45a"). InnerVolumeSpecName "kube-api-access-tjmvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:23.720876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.720843 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:23.720876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:23.720871 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjmvc\" (UniqueName: \"kubernetes.io/projected/f19ce1e1-ff8f-4c92-bcce-404325e5f45a-kube-api-access-tjmvc\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:24.430845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:24.430814 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" Oct 30 22:07:24.431056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:24.430849 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz" event={"ID":"f19ce1e1-ff8f-4c92-bcce-404325e5f45a","Type":"ContainerDied","Data":"a3adf764a72895d72482ab216f38e8518468d05b175761b77d0b6c19a68f37c9"} Oct 30 22:07:24.431056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:24.430929 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3adf764a72895d72482ab216f38e8518468d05b175761b77d0b6c19a68f37c9" Oct 30 22:07:26.137510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:26.137476 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:26.151573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:26.151553 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:26.457555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:26.457525 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:07:30.729977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.729924 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs"] Oct 30 22:07:30.730508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.730493 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f19ce1e1-ff8f-4c92-bcce-404325e5f45a" containerName="revoke-keys" Oct 30 22:07:30.730556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.730510 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19ce1e1-ff8f-4c92-bcce-404325e5f45a" containerName="revoke-keys" Oct 30 22:07:30.730628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.730619 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19ce1e1-ff8f-4c92-bcce-404325e5f45a" containerName="revoke-keys" Oct 30 22:07:30.734279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.734266 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.839327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.839286 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:30.839512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.839405 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:07:30.842699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.842676 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs"] Oct 30 22:07:30.859146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.859117 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2ph\" (UniqueName: \"kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.859280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.859260 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.859336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.859324 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.961004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.960966 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.961326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.961098 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.961326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.961293 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2ph\" (UniqueName: \"kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.962657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.962632 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:30.964224 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:30.964204 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:31.024588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:31.024536 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2ph\" (UniqueName: \"kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph\") pod \"maas-api-revoke-sub-77d80222-jqdgs\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:31.053320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:31.053300 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:31.065211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:31.065194 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6"] Oct 30 22:07:31.069678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:31.069660 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d532-pcts6"] Oct 30 22:07:31.425521 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:31.425488 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d133f40_b85d_4929_8e2a_f1182a4bd9ac.slice/crio-16d60d42319d2a7abf6924591edc80b71cee1a71f3a706b488485773f83572b7 WatchSource:0}: Error finding container 16d60d42319d2a7abf6924591edc80b71cee1a71f3a706b488485773f83572b7: Status 404 returned error can't find the container with id 16d60d42319d2a7abf6924591edc80b71cee1a71f3a706b488485773f83572b7 Oct 30 22:07:31.429040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:31.429016 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs"] Oct 30 22:07:31.502146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:31.502121 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" event={"ID":"3d133f40-b85d-4929-8e2a-f1182a4bd9ac","Type":"ContainerStarted","Data":"16d60d42319d2a7abf6924591edc80b71cee1a71f3a706b488485773f83572b7"} Oct 30 22:07:32.511983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:32.511926 5607 generic.go:356] "Generic (PLEG): container finished" podID="3d133f40-b85d-4929-8e2a-f1182a4bd9ac" containerID="652ee63540357d6adf9a182422187a3c34c7adfa43bd320aee5c51d76bca09fd" exitCode=0 Oct 30 22:07:32.512384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:32.511991 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" event={"ID":"3d133f40-b85d-4929-8e2a-f1182a4bd9ac","Type":"ContainerDied","Data":"652ee63540357d6adf9a182422187a3c34c7adfa43bd320aee5c51d76bca09fd"} Oct 30 22:07:32.870676 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:32.870642 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9a25bf-6af5-4690-b0ed-48d5522566b2" path="/var/lib/kubelet/pods/0e9a25bf-6af5-4690-b0ed-48d5522566b2/volumes" Oct 30 22:07:33.659327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.659303 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:33.793385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.793303 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca\") pod \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " Oct 30 22:07:33.793711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.793673 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca" pod "3d133f40-b85d-4929-8e2a-f1182a4bd9ac" (UID: "3d133f40-b85d-4929-8e2a-f1182a4bd9ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:33.793937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.793911 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca\") pod \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " Oct 30 22:07:33.794099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.794081 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph\" (UniqueName: \"kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph\") pod \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\" (UID: \"3d133f40-b85d-4929-8e2a-f1182a4bd9ac\") " Oct 30 22:07:33.794641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.794623 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:33.797622 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.797593 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph" pod "3d133f40-b85d-4929-8e2a-f1182a4bd9ac" (UID: "3d133f40-b85d-4929-8e2a-f1182a4bd9ac"). InnerVolumeSpecName "kube-api-access-6b2ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:33.797725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.797672 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca" pod "3d133f40-b85d-4929-8e2a-f1182a4bd9ac" (UID: "3d133f40-b85d-4929-8e2a-f1182a4bd9ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:33.896355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.896330 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:33.896355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:33.896355 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b2ph\" (UniqueName: \"kubernetes.io/projected/3d133f40-b85d-4929-8e2a-f1182a4bd9ac-kube-api-access-6b2ph\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:34.543351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:34.543321 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" Oct 30 22:07:34.543546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:34.543400 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs" event={"ID":"3d133f40-b85d-4929-8e2a-f1182a4bd9ac","Type":"ContainerDied","Data":"16d60d42319d2a7abf6924591edc80b71cee1a71f3a706b488485773f83572b7"} Oct 30 22:07:34.543546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:34.543486 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16d60d42319d2a7abf6924591edc80b71cee1a71f3a706b488485773f83572b7" Oct 30 22:07:42.399724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.399688 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg"] Oct 30 22:07:42.400340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.400322 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3d133f40-b85d-4929-8e2a-f1182a4bd9ac" containerName="revoke-keys" Oct 30 22:07:42.400381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.400341 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d133f40-b85d-4929-8e2a-f1182a4bd9ac" containerName="revoke-keys" Oct 30 22:07:42.400473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.400463 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d133f40-b85d-4929-8e2a-f1182a4bd9ac" containerName="revoke-keys" Oct 30 22:07:42.404260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.404244 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.458660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.458634 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:42.481659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.481629 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.481800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.481681 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.481800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.481749 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5vh\" (UniqueName: \"kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.509715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.509694 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:07:42.512916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.512896 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg"] Oct 30 22:07:42.582613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.582587 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.582730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.582621 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.582730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.582641 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5vh\" (UniqueName: \"kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.583398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.583371 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.585144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.585121 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.645376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.645349 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5vh\" (UniqueName: \"kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh\") pod \"maas-api-revoke-sub-6d646474-9r2qg\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:42.718153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:42.718127 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:43.091939 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:43.091910 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd10f42a_c43e_4b98_a0dc_5e9e2e507389.slice/crio-75e27d8c3eb4bfa4f87ccd4e3ea420e389102de1cb1a90991aa8f092ed14c81f WatchSource:0}: Error finding container 75e27d8c3eb4bfa4f87ccd4e3ea420e389102de1cb1a90991aa8f092ed14c81f: Status 404 returned error can't find the container with id 75e27d8c3eb4bfa4f87ccd4e3ea420e389102de1cb1a90991aa8f092ed14c81f Oct 30 22:07:43.096227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:43.096153 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg"] Oct 30 22:07:43.612424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:43.612388 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" event={"ID":"fd10f42a-c43e-4b98-a0dc-5e9e2e507389","Type":"ContainerStarted","Data":"a6df4fe67246553d025f623d7516ce66692350babef6d37dd787ed28adbf45d7"} Oct 30 22:07:43.612799 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:43.612450 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" event={"ID":"fd10f42a-c43e-4b98-a0dc-5e9e2e507389","Type":"ContainerStarted","Data":"75e27d8c3eb4bfa4f87ccd4e3ea420e389102de1cb1a90991aa8f092ed14c81f"} Oct 30 22:07:43.732784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:43.732738 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" podStartSLOduration=1.7327247639999999 podStartE2EDuration="1.732724764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:43.731620858 +0000 UTC m=+1243.132176981" watchObservedRunningTime="2026-10-30 22:07:43.732724764 +0000 UTC m=+1243.133280930" Oct 30 22:07:44.620241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:44.620206 5607 generic.go:356] "Generic (PLEG): container finished" podID="fd10f42a-c43e-4b98-a0dc-5e9e2e507389" containerID="a6df4fe67246553d025f623d7516ce66692350babef6d37dd787ed28adbf45d7" exitCode=0 Oct 30 22:07:44.620619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:44.620262 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" event={"ID":"fd10f42a-c43e-4b98-a0dc-5e9e2e507389","Type":"ContainerDied","Data":"a6df4fe67246553d025f623d7516ce66692350babef6d37dd787ed28adbf45d7"} Oct 30 22:07:46.673155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.672080 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 22:07:46.691904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.691801 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" event={"ID":"fd10f42a-c43e-4b98-a0dc-5e9e2e507389","Type":"ContainerDied","Data":"75e27d8c3eb4bfa4f87ccd4e3ea420e389102de1cb1a90991aa8f092ed14c81f"} Oct 30 22:07:46.691904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.691863 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75e27d8c3eb4bfa4f87ccd4e3ea420e389102de1cb1a90991aa8f092ed14c81f" Oct 30 22:07:46.702838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.702542 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:46.802982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.802055 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 22:07:46.802982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.802871 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:46.862034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.860428 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca\") pod \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " Oct 30 22:07:46.862034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.860604 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca\") pod \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " Oct 30 22:07:46.862034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.860674 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh\" (UniqueName: \"kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh\") pod \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\" (UID: \"fd10f42a-c43e-4b98-a0dc-5e9e2e507389\") " Oct 30 22:07:46.990979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.966341 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca" pod "fd10f42a-c43e-4b98-a0dc-5e9e2e507389" (UID: "fd10f42a-c43e-4b98-a0dc-5e9e2e507389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:46.990979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:46.978058 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:47.062082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.062033 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh" pod "fd10f42a-c43e-4b98-a0dc-5e9e2e507389" (UID: "fd10f42a-c43e-4b98-a0dc-5e9e2e507389"). InnerVolumeSpecName "kube-api-access-7z5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:47.098158 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.070034 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:47.098158 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.070061 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z5vh\" (UniqueName: \"kubernetes.io/projected/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-kube-api-access-7z5vh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:47.126238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.126170 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca" pod "fd10f42a-c43e-4b98-a0dc-5e9e2e507389" (UID: "fd10f42a-c43e-4b98-a0dc-5e9e2e507389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:47.144087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.144049 5607 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:47.179216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.179176 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd10f42a-c43e-4b98-a0dc-5e9e2e507389-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:47.699439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:47.699411 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg" Oct 30 22:07:49.081457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:49.081424 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr"] Oct 30 22:07:49.088907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:49.088880 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3002d34b-8pnzr"] Oct 30 22:07:50.333678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.333650 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p"] Oct 30 22:07:50.334319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.334301 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fd10f42a-c43e-4b98-a0dc-5e9e2e507389" containerName="revoke-keys" Oct 30 22:07:50.334386 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.334323 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd10f42a-c43e-4b98-a0dc-5e9e2e507389" containerName="revoke-keys" Oct 30 22:07:50.334457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.334446 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd10f42a-c43e-4b98-a0dc-5e9e2e507389" containerName="revoke-keys" Oct 30 22:07:50.338057 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.338038 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.392318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.392299 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:50.415576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.415553 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.415681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.415586 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.415681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.415606 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5l8\" (UniqueName: \"kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.442663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.442640 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:07:50.448395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.448373 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p"] Oct 30 22:07:50.516713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.516681 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.516856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.516726 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.516856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.516757 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5l8\" (UniqueName: \"kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.517463 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.517439 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.519616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.519596 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.580045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.580018 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5l8\" (UniqueName: \"kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8\") pod \"maas-api-revoke-sub-dcd92762-dxv2p\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.651593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.651536 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:50.878508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:50.878475 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc0c432-bf3d-45d0-8c06-8da3bb4fb587" path="/var/lib/kubelet/pods/bdc0c432-bf3d-45d0-8c06-8da3bb4fb587/volumes" Oct 30 22:07:51.024084 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:51.024054 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5fecfd_4e66_4517_95bb_beff04eb037e.slice/crio-719d3b198aea981aade12d4cdb6c77d80ecef8b390e68ef39e6cad2d88f0f8c6 WatchSource:0}: Error finding container 719d3b198aea981aade12d4cdb6c77d80ecef8b390e68ef39e6cad2d88f0f8c6: Status 404 returned error can't find the container with id 719d3b198aea981aade12d4cdb6c77d80ecef8b390e68ef39e6cad2d88f0f8c6 Oct 30 22:07:51.028352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:51.028307 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p"] Oct 30 22:07:51.738759 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:51.738725 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" event={"ID":"2e5fecfd-4e66-4517-95bb-beff04eb037e","Type":"ContainerStarted","Data":"e713dbc7e4422006ecd42a0ff4250a5f6a82486af09188074e02386a7051ff8b"} Oct 30 22:07:51.739167 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:51.738786 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" event={"ID":"2e5fecfd-4e66-4517-95bb-beff04eb037e","Type":"ContainerStarted","Data":"719d3b198aea981aade12d4cdb6c77d80ecef8b390e68ef39e6cad2d88f0f8c6"} Oct 30 22:07:51.861724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:51.861676 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" podStartSLOduration=1.861661798 podStartE2EDuration="1.861661798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:51.85898718 +0000 UTC m=+1251.259543301" watchObservedRunningTime="2026-10-30 22:07:51.861661798 +0000 UTC m=+1251.262217919" Oct 30 22:07:52.748567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:52.748533 5607 generic.go:356] "Generic (PLEG): container finished" podID="2e5fecfd-4e66-4517-95bb-beff04eb037e" containerID="e713dbc7e4422006ecd42a0ff4250a5f6a82486af09188074e02386a7051ff8b" exitCode=0 Oct 30 22:07:52.749038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:52.748583 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" event={"ID":"2e5fecfd-4e66-4517-95bb-beff04eb037e","Type":"ContainerDied","Data":"e713dbc7e4422006ecd42a0ff4250a5f6a82486af09188074e02386a7051ff8b"} Oct 30 22:07:52.863589 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:07:52.863550 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:07:53.904990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.904964 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:53.955728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.955701 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca\") pod \"2e5fecfd-4e66-4517-95bb-beff04eb037e\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " Oct 30 22:07:53.955848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.955748 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8\" (UniqueName: \"kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8\") pod \"2e5fecfd-4e66-4517-95bb-beff04eb037e\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " Oct 30 22:07:53.955848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.955814 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca\") pod \"2e5fecfd-4e66-4517-95bb-beff04eb037e\" (UID: \"2e5fecfd-4e66-4517-95bb-beff04eb037e\") " Oct 30 22:07:53.956244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.956218 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca" pod "2e5fecfd-4e66-4517-95bb-beff04eb037e" (UID: "2e5fecfd-4e66-4517-95bb-beff04eb037e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:53.956494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.956478 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:53.958297 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.958272 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca" pod "2e5fecfd-4e66-4517-95bb-beff04eb037e" (UID: "2e5fecfd-4e66-4517-95bb-beff04eb037e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:53.958472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:53.958453 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8" pod "2e5fecfd-4e66-4517-95bb-beff04eb037e" (UID: "2e5fecfd-4e66-4517-95bb-beff04eb037e"). InnerVolumeSpecName "kube-api-access-6q5l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:54.057726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.057661 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5fecfd-4e66-4517-95bb-beff04eb037e-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:54.057726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.057689 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q5l8\" (UniqueName: \"kubernetes.io/projected/2e5fecfd-4e66-4517-95bb-beff04eb037e-kube-api-access-6q5l8\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:54.426390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.426317 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7"] Oct 30 22:07:54.426921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.426906 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2e5fecfd-4e66-4517-95bb-beff04eb037e" containerName="revoke-keys" Oct 30 22:07:54.427010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.426924 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5fecfd-4e66-4517-95bb-beff04eb037e" containerName="revoke-keys" Oct 30 22:07:54.427068 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.427030 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5fecfd-4e66-4517-95bb-beff04eb037e" containerName="revoke-keys" Oct 30 22:07:54.430449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.430435 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.539522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.539489 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7"] Oct 30 22:07:54.563992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.563939 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564063 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfbd\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564105 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564185 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564262 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564309 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564466 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564552 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564501 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.564618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.564602 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.665521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665486 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.665771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665601 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.665771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665688 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.665771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665714 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.665771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665762 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.666106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665883 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.666106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665907 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfbd\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.666106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.665936 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.666647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.666625 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.683042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.675544 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.683042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.675800 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.683042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.676113 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.683042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.676339 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.683042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.681523 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.683042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.682186 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.691615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.691587 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.738010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.737981 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.738922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.738880 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfbd\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.745525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.745501 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:54.766287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.766259 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" event={"ID":"2e5fecfd-4e66-4517-95bb-beff04eb037e","Type":"ContainerDied","Data":"719d3b198aea981aade12d4cdb6c77d80ecef8b390e68ef39e6cad2d88f0f8c6"} Oct 30 22:07:54.766426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.766306 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719d3b198aea981aade12d4cdb6c77d80ecef8b390e68ef39e6cad2d88f0f8c6" Oct 30 22:07:54.766426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:54.766305 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p" Oct 30 22:07:55.173363 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:55.173302 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b4a034_13ec_4e36_9a62_17c4aa5008e0.slice/crio-6258be61e3b174750516d772ea4dcfe7263bd99e138e1a524f0741740620a144 WatchSource:0}: Error finding container 6258be61e3b174750516d772ea4dcfe7263bd99e138e1a524f0741740620a144: Status 404 returned error can't find the container with id 6258be61e3b174750516d772ea4dcfe7263bd99e138e1a524f0741740620a144 Oct 30 22:07:55.175907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.175870 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:55.176067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.175947 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:55.176067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.176003 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:07:55.177444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.177422 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7"] Oct 30 22:07:55.785080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.785046 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" event={"ID":"76b4a034-13ec-4e36-9a62-17c4aa5008e0","Type":"ContainerStarted","Data":"639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb"} Oct 30 22:07:55.785285 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.785097 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" event={"ID":"76b4a034-13ec-4e36-9a62-17c4aa5008e0","Type":"ContainerStarted","Data":"6258be61e3b174750516d772ea4dcfe7263bd99e138e1a524f0741740620a144"} Oct 30 22:07:55.893781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.893754 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-74pp2"] Oct 30 22:07:55.924807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.924781 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:55.941817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.941777 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podStartSLOduration=1.9417648490000001 podStartE2EDuration="1.941764849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:55.917434099 +0000 UTC m=+1255.317990220" watchObservedRunningTime="2026-10-30 22:07:55.941764849 +0000 UTC m=+1255.342320970" Oct 30 22:07:55.957478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.957451 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-74pp2"] Oct 30 22:07:55.994100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.994071 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:55.994202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.994106 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:55.994202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.994124 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:55.994285 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:55.994237 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lgg\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.120030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.116649 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lgg\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.120030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.116989 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.120030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.117024 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.120030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.117056 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.120030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.117934 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.120030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.118696 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.122649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.122621 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.181833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.181801 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lgg\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg\") pod \"payload-processing-external-model-6844c89498-74pp2\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.183703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.183682 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr"] Oct 30 22:07:56.188179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.188162 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.239426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.239400 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:56.243147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.243128 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:56.243242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.243186 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:07:56.258703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.258676 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr"] Oct 30 22:07:56.329269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.328704 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.329269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.328891 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pvz\" (UniqueName: \"kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.329269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.328923 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.430864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.430760 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pvz\" (UniqueName: \"kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.430864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.430803 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.431094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.430999 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.431678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.431654 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.433490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.433470 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.521557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.520822 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pvz\" (UniqueName: \"kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz\") pod \"maas-api-revoke-sub-48dd9313-7h4mr\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.532159 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.531931 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:07:56.692362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.690611 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-74pp2"] Oct 30 22:07:56.748047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.747040 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:07:56.748449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.748341 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:07:56.748449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.748394 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 22:07:56.822806 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.822645 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" event={"ID":"7980f00e-1872-47e6-83f0-3d231df70d77","Type":"ContainerStarted","Data":"cfbe2d3d7f44b71ab0c27af7112ee21adf6ade22697b6a548d61678b58004a0e"} Oct 30 22:07:56.988108 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:07:56.988058 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e627bb5_1d9e_47ed_bc1a_90f5fd5a682a.slice/crio-3789c1c48dda2402a2ccecc0099d35efe41623fa425cbc318b64633dac4793f5 WatchSource:0}: Error finding container 3789c1c48dda2402a2ccecc0099d35efe41623fa425cbc318b64633dac4793f5: Status 404 returned error can't find the container with id 3789c1c48dda2402a2ccecc0099d35efe41623fa425cbc318b64633dac4793f5 Oct 30 22:07:56.992345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:56.992214 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr"] Oct 30 22:07:57.746693 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.746659 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:07:57.747098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.746709 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 22:07:57.832251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.832173 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" event={"ID":"7980f00e-1872-47e6-83f0-3d231df70d77","Type":"ContainerStarted","Data":"5b0fb8370793d74da4f2f44987e2d91c409790eee317b2b6195eeb6286f3f5a5"} Oct 30 22:07:57.832499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.832478 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:07:57.834378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.834338 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" event={"ID":"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a","Type":"ContainerStarted","Data":"db1a5b83f958e08f51c9b12dfed1fe6b22e2bd0b121d4bb5850d3e5b473f02b5"} Oct 30 22:07:57.834508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.834390 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" event={"ID":"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a","Type":"ContainerStarted","Data":"3789c1c48dda2402a2ccecc0099d35efe41623fa425cbc318b64633dac4793f5"} Oct 30 22:07:57.954425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:57.954376 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" podStartSLOduration=2.954363023 podStartE2EDuration="2.954363023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:57.951945353 +0000 UTC m=+1257.352501473" watchObservedRunningTime="2026-10-30 22:07:57.954363023 +0000 UTC m=+1257.354919143" Oct 30 22:07:58.083605 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:58.083504 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" podStartSLOduration=2.083490789 podStartE2EDuration="2.083490789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:58.082847142 +0000 UTC m=+1257.483403276" watchObservedRunningTime="2026-10-30 22:07:58.083490789 +0000 UTC m=+1257.484046909" Oct 30 22:07:58.746195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:58.746161 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:07:58.746375 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:58.746209 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 22:07:58.843131 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:58.843094 5607 generic.go:356] "Generic (PLEG): container finished" podID="8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" containerID="db1a5b83f958e08f51c9b12dfed1fe6b22e2bd0b121d4bb5850d3e5b473f02b5" exitCode=0 Oct 30 22:07:58.843583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:58.843176 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" event={"ID":"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a","Type":"ContainerDied","Data":"db1a5b83f958e08f51c9b12dfed1fe6b22e2bd0b121d4bb5850d3e5b473f02b5"} Oct 30 22:07:59.746546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:59.746516 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:07:59.746705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:07:59.746562 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 22:08:00.050471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.050448 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:08:00.091605 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.091582 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz\" (UniqueName: \"kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz\") pod \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " Oct 30 22:08:00.091722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.091686 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca\") pod \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " Oct 30 22:08:00.091771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.091738 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca\") pod \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\" (UID: \"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a\") " Oct 30 22:08:00.092055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.092033 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca" pod "8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" (UID: "8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:00.092284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.092231 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:00.093745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.093718 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz" pod "8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" (UID: "8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a"). InnerVolumeSpecName "kube-api-access-72pvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:00.093886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.093869 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca" pod "8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" (UID: "8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:00.193508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.193481 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:00.193508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.193506 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pvz\" (UniqueName: \"kubernetes.io/projected/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a-kube-api-access-72pvz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:00.746726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.746683 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:08:00.746904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.746737 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 22:08:00.859493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.859469 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" Oct 30 22:08:00.859655 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.859495 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr" event={"ID":"8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a","Type":"ContainerDied","Data":"3789c1c48dda2402a2ccecc0099d35efe41623fa425cbc318b64633dac4793f5"} Oct 30 22:08:00.859655 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:00.859563 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3789c1c48dda2402a2ccecc0099d35efe41623fa425cbc318b64633dac4793f5" Oct 30 22:08:01.081887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.081811 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg"] Oct 30 22:08:01.082423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.082409 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" containerName="revoke-keys" Oct 30 22:08:01.082475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.082424 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" containerName="revoke-keys" Oct 30 22:08:01.082538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.082529 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" containerName="revoke-keys" Oct 30 22:08:01.086156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.086142 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7"] Oct 30 22:08:01.086274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.086257 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.090522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.090505 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.140212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.140145 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:01.140212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.140178 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:08:01.147385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.144096 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg"] Oct 30 22:08:01.187095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.187070 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7"] Oct 30 22:08:01.227210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.227181 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcrf\" (UniqueName: \"kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.227373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.227248 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.227373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.227363 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swx85\" (UniqueName: \"kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.227470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.227405 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.227525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.227512 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.227557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.227538 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.328524 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.328487 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swx85\" (UniqueName: \"kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.328685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.328546 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.328685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.328659 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.328827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.328691 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.328873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.328827 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcrf\" (UniqueName: \"kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.328918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.328906 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.329622 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.329595 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.329867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.329843 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.331244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.331218 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.331338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.331247 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.391221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.391169 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcrf\" (UniqueName: \"kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf\") pod \"maas-api-revoke-sub-ac9ef690-zqkxg\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.391321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.391255 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swx85\" (UniqueName: \"kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85\") pod \"maas-api-revoke-sub-40ec7da7-lplb7\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.427834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.427804 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:01.435493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.435473 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:01.746881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.746846 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:08:01.747149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.746914 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 22:08:01.820788 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:01.820757 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7a9432_99ab_46c9_a2eb_061b13d0548d.slice/crio-f3248a3d48781b35b86f8d8d3f7f49ad65be63720b7282bdf3cce9b4d4650a68 WatchSource:0}: Error finding container f3248a3d48781b35b86f8d8d3f7f49ad65be63720b7282bdf3cce9b4d4650a68: Status 404 returned error can't find the container with id f3248a3d48781b35b86f8d8d3f7f49ad65be63720b7282bdf3cce9b4d4650a68 Oct 30 22:08:01.823946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.823926 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7"] Oct 30 22:08:01.867543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.867513 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" event={"ID":"3f7a9432-99ab-46c9-a2eb-061b13d0548d","Type":"ContainerStarted","Data":"f3248a3d48781b35b86f8d8d3f7f49ad65be63720b7282bdf3cce9b4d4650a68"} Oct 30 22:08:01.871022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:01.871002 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg"] Oct 30 22:08:02.751251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.751223 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:08:02.751724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.751670 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:08:02.752314 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.752299 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:08:02.927220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.927155 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" event={"ID":"8d466542-974b-4bb6-9d66-5985588eca3e","Type":"ContainerStarted","Data":"6bfa92c0dacf0faae5bf89691a12a272bbede97a133d2fceea620faf91a3bad2"} Oct 30 22:08:02.928046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.928019 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" event={"ID":"8d466542-974b-4bb6-9d66-5985588eca3e","Type":"ContainerStarted","Data":"5d849df2075542fd118df15ee42f40487b052f963db5c5284d594ebbc1791ea3"} Oct 30 22:08:02.935999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.935074 5607 generic.go:356] "Generic (PLEG): container finished" podID="3f7a9432-99ab-46c9-a2eb-061b13d0548d" containerID="a4834c4b4eba86691999f26611b5ebb2f7dfe579e154ba6d3d6deaefe0326976" exitCode=0 Oct 30 22:08:02.935999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:02.935301 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" event={"ID":"3f7a9432-99ab-46c9-a2eb-061b13d0548d","Type":"ContainerDied","Data":"a4834c4b4eba86691999f26611b5ebb2f7dfe579e154ba6d3d6deaefe0326976"} Oct 30 22:08:03.512976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:03.505999 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" podStartSLOduration=2.446593195 podStartE2EDuration="2.446593195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:03.337297059 +0000 UTC m=+1262.737853181" watchObservedRunningTime="2026-10-30 22:08:03.446593195 +0000 UTC m=+1262.847149317" Oct 30 22:08:04.031971 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.030494 5607 generic.go:356] "Generic (PLEG): container finished" podID="8d466542-974b-4bb6-9d66-5985588eca3e" containerID="6bfa92c0dacf0faae5bf89691a12a272bbede97a133d2fceea620faf91a3bad2" exitCode=0 Oct 30 22:08:04.031971 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.031720 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" event={"ID":"8d466542-974b-4bb6-9d66-5985588eca3e","Type":"ContainerDied","Data":"6bfa92c0dacf0faae5bf89691a12a272bbede97a133d2fceea620faf91a3bad2"} Oct 30 22:08:04.064341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.064303 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw"] Oct 30 22:08:04.091092 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.091044 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d05022b-f2tzw"] Oct 30 22:08:04.651996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.631412 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Oct 30 22:08:04.651996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.631774 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 22:08:04.670106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.669659 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:04.894030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.888223 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 22:08:05.040011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:04.976721 5607 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": context deadline exceeded" Oct 30 22:08:05.053106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.053065 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:05.151754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.138870 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1143ae-4dc6-4156-b1ea-52d1dfb547c9" path="/var/lib/kubelet/pods/af1143ae-4dc6-4156-b1ea-52d1dfb547c9/volumes" Oct 30 22:08:05.218902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.218297 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:05.223709 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.223390 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:05.251976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.247717 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca\") pod \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " Oct 30 22:08:05.251976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.247778 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca\") pod \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " Oct 30 22:08:05.251976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.248000 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85\" (UniqueName: \"kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85\") pod \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\" (UID: \"3f7a9432-99ab-46c9-a2eb-061b13d0548d\") " Oct 30 22:08:05.269649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.268203 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca" pod "3f7a9432-99ab-46c9-a2eb-061b13d0548d" (UID: "3f7a9432-99ab-46c9-a2eb-061b13d0548d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:05.273981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.271515 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:05.273981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.271565 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:05.279041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.278993 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca" pod "3f7a9432-99ab-46c9-a2eb-061b13d0548d" (UID: "3f7a9432-99ab-46c9-a2eb-061b13d0548d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:05.286462 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.286421 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85" pod "3f7a9432-99ab-46c9-a2eb-061b13d0548d" (UID: "3f7a9432-99ab-46c9-a2eb-061b13d0548d"). InnerVolumeSpecName "kube-api-access-swx85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:05.387987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.375198 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swx85\" (UniqueName: \"kubernetes.io/projected/3f7a9432-99ab-46c9-a2eb-061b13d0548d-kube-api-access-swx85\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:05.387987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.375231 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:05.387987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.375249 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7a9432-99ab-46c9-a2eb-061b13d0548d-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:05.697366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.688442 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:05.697366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.695853 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Oct 30 22:08:05.697366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.696564 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:08:05.697366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.696622 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:08:05.805171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:05.788834 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:08:06.082790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.081164 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:08:06.178627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.178584 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:06.195297 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.195255 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" event={"ID":"8d466542-974b-4bb6-9d66-5985588eca3e","Type":"ContainerDied","Data":"5d849df2075542fd118df15ee42f40487b052f963db5c5284d594ebbc1791ea3"} Oct 30 22:08:06.195487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.195322 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d849df2075542fd118df15ee42f40487b052f963db5c5284d594ebbc1791ea3" Oct 30 22:08:06.195487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.195396 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg" Oct 30 22:08:06.198181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.198158 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" Oct 30 22:08:06.199853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.198439 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7" event={"ID":"3f7a9432-99ab-46c9-a2eb-061b13d0548d","Type":"ContainerDied","Data":"f3248a3d48781b35b86f8d8d3f7f49ad65be63720b7282bdf3cce9b4d4650a68"} Oct 30 22:08:06.200050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.200034 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3248a3d48781b35b86f8d8d3f7f49ad65be63720b7282bdf3cce9b4d4650a68" Oct 30 22:08:06.334497 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.334405 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca\") pod \"8d466542-974b-4bb6-9d66-5985588eca3e\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " Oct 30 22:08:06.335181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.335060 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca" pod "8d466542-974b-4bb6-9d66-5985588eca3e" (UID: "8d466542-974b-4bb6-9d66-5985588eca3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:06.335676 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.335657 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca\") pod \"8d466542-974b-4bb6-9d66-5985588eca3e\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " Oct 30 22:08:06.335936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.335919 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf\" (UniqueName: \"kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf\") pod \"8d466542-974b-4bb6-9d66-5985588eca3e\" (UID: \"8d466542-974b-4bb6-9d66-5985588eca3e\") " Oct 30 22:08:06.337054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.337031 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:06.362095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.362058 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf" pod "8d466542-974b-4bb6-9d66-5985588eca3e" (UID: "8d466542-974b-4bb6-9d66-5985588eca3e"). InnerVolumeSpecName "kube-api-access-krcrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:06.362279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.362237 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca" pod "8d466542-974b-4bb6-9d66-5985588eca3e" (UID: "8d466542-974b-4bb6-9d66-5985588eca3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:06.438641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.438571 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcrf\" (UniqueName: \"kubernetes.io/projected/8d466542-974b-4bb6-9d66-5985588eca3e-kube-api-access-krcrf\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:06.438641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:06.438614 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466542-974b-4bb6-9d66-5985588eca3e-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:08.825882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:08.825837 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59fbf69455-5km7v"] Oct 30 22:08:08.846364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:08.846329 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:08:08.879321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:08.879280 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-74pp2"] Oct 30 22:08:08.880871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:08.880840 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:08:09.616191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.616150 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p"] Oct 30 22:08:09.617363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.617337 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3f7a9432-99ab-46c9-a2eb-061b13d0548d" containerName="revoke-keys" Oct 30 22:08:09.617531 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.617518 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7a9432-99ab-46c9-a2eb-061b13d0548d" containerName="revoke-keys" Oct 30 22:08:09.617658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.617647 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8d466542-974b-4bb6-9d66-5985588eca3e" containerName="revoke-keys" Oct 30 22:08:09.617741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.617730 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d466542-974b-4bb6-9d66-5985588eca3e" containerName="revoke-keys" Oct 30 22:08:09.618018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.618005 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7a9432-99ab-46c9-a2eb-061b13d0548d" containerName="revoke-keys" Oct 30 22:08:09.618144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.618132 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d466542-974b-4bb6-9d66-5985588eca3e" containerName="revoke-keys" Oct 30 22:08:09.646296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.646270 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.686759 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.686727 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p"] Oct 30 22:08:09.691939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.691904 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.715464 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715431 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.715654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715486 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.715654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715619 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.715654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715652 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.715829 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715710 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.716036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715876 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trtk\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.716036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715945 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.716036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.715997 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.817833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.817774 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trtk\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.818898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.818866 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819105 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819188 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819240 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819264 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819374 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819397 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819417 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.819517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.819451 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.820353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.820329 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.821165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.821122 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.821254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.821187 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.821527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.821498 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.826053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.826008 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.827529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.827505 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.882291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.882212 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trtk\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.883219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.883201 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:09.965390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:09.965366 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:10.387932 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:10.387900 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb1adb5_7c81_4647_87a4_cd3f2820c2c5.slice/crio-2b8da06fd553e75700765a7fd0db62f58c7df44ae31013fe41e7e8ce3bca5797 WatchSource:0}: Error finding container 2b8da06fd553e75700765a7fd0db62f58c7df44ae31013fe41e7e8ce3bca5797: Status 404 returned error can't find the container with id 2b8da06fd553e75700765a7fd0db62f58c7df44ae31013fe41e7e8ce3bca5797 Oct 30 22:08:10.390446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:10.390407 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:10.390561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:10.390492 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:10.390561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:10.390541 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:10.392362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:10.392339 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p"] Oct 30 22:08:11.234745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:11.234707 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" event={"ID":"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5","Type":"ContainerStarted","Data":"0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7"} Oct 30 22:08:11.235185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:11.234768 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" event={"ID":"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5","Type":"ContainerStarted","Data":"2b8da06fd553e75700765a7fd0db62f58c7df44ae31013fe41e7e8ce3bca5797"} Oct 30 22:08:11.965845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:11.965819 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:11.967095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:11.967071 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:08:11.967217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:11.967117 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:08:12.965746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:12.965712 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:08:12.966206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:12.965765 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:08:13.826845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:13.826805 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" podUID="4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" containerName="payload-processing" containerID="cri-o://12cb02e039f9b6cbab66d0a81749500b98da06eed689e0b4db1e85ab0ea2dffd" gracePeriod=25 Oct 30 22:08:13.880670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:13.880633 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" containerName="payload-processing" containerID="cri-o://5b0fb8370793d74da4f2f44987e2d91c409790eee317b2b6195eeb6286f3f5a5" gracePeriod=25 Oct 30 22:08:13.965672 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:13.965645 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:08:13.965777 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:13.965700 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:08:14.257852 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.257825 5607 generic.go:356] "Generic (PLEG): container finished" podID="7980f00e-1872-47e6-83f0-3d231df70d77" containerID="5b0fb8370793d74da4f2f44987e2d91c409790eee317b2b6195eeb6286f3f5a5" exitCode=0 Oct 30 22:08:14.258014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.257938 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" event={"ID":"7980f00e-1872-47e6-83f0-3d231df70d77","Type":"ContainerDied","Data":"5b0fb8370793d74da4f2f44987e2d91c409790eee317b2b6195eeb6286f3f5a5"} Oct 30 22:08:14.258198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.258176 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" event={"ID":"7980f00e-1872-47e6-83f0-3d231df70d77","Type":"ContainerDied","Data":"cfbe2d3d7f44b71ab0c27af7112ee21adf6ade22697b6a548d61678b58004a0e"} Oct 30 22:08:14.258283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.258214 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfbe2d3d7f44b71ab0c27af7112ee21adf6ade22697b6a548d61678b58004a0e" Oct 30 22:08:14.259641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.259620 5607 generic.go:356] "Generic (PLEG): container finished" podID="4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" containerID="12cb02e039f9b6cbab66d0a81749500b98da06eed689e0b4db1e85ab0ea2dffd" exitCode=0 Oct 30 22:08:14.259799 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.259782 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" event={"ID":"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51","Type":"ContainerDied","Data":"12cb02e039f9b6cbab66d0a81749500b98da06eed689e0b4db1e85ab0ea2dffd"} Oct 30 22:08:14.259876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.259827 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" event={"ID":"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51","Type":"ContainerDied","Data":"df07660bb5c250b6153730690ebb5189139bfa1af86692eb2a64073529e74a15"} Oct 30 22:08:14.259876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.259852 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df07660bb5c250b6153730690ebb5189139bfa1af86692eb2a64073529e74a15" Oct 30 22:08:14.268509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.268488 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:08:14.271639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.271620 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:08:14.288683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.288664 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6\") pod \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " Oct 30 22:08:14.288791 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.288759 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume\") pod \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " Oct 30 22:08:14.288854 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.288828 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials\") pod \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " Oct 30 22:08:14.288941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.288925 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume\") pod \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\" (UID: \"4a3c86cf-55b9-4ecc-ab77-654a56fb1a51\") " Oct 30 22:08:14.289123 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.289100 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume" pod "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" (UID: "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:14.289362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.289335 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume" pod "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" (UID: "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:14.289642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.289623 5607 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-routing-overlay-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.289708 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.289648 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.290999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.290975 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6" pod "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" (UID: "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51"). InnerVolumeSpecName "kube-api-access-slth6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:14.291148 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.291104 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials" pod "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" (UID: "4a3c86cf-55b9-4ecc-ab77-654a56fb1a51"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:14.390225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.390197 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume\") pod \"7980f00e-1872-47e6-83f0-3d231df70d77\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " Oct 30 22:08:14.390381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.390317 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume\") pod \"7980f00e-1872-47e6-83f0-3d231df70d77\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " Oct 30 22:08:14.390455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.390388 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials\") pod \"7980f00e-1872-47e6-83f0-3d231df70d77\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " Oct 30 22:08:14.390560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.390542 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg\") pod \"7980f00e-1872-47e6-83f0-3d231df70d77\" (UID: \"7980f00e-1872-47e6-83f0-3d231df70d77\") " Oct 30 22:08:14.390657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.390634 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume" pod "7980f00e-1872-47e6-83f0-3d231df70d77" (UID: "7980f00e-1872-47e6-83f0-3d231df70d77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:14.390711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.390680 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume" pod "7980f00e-1872-47e6-83f0-3d231df70d77" (UID: "7980f00e-1872-47e6-83f0-3d231df70d77"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:14.391197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.391123 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.391197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.391150 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slth6\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-kube-api-access-slth6\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.391197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.391163 5607 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7980f00e-1872-47e6-83f0-3d231df70d77-routing-overlay-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.391197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.391174 5607 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51-provider-credentials\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.393045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.392991 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg" pod "7980f00e-1872-47e6-83f0-3d231df70d77" (UID: "7980f00e-1872-47e6-83f0-3d231df70d77"). InnerVolumeSpecName "kube-api-access-95lgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:14.393045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.393023 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials" pod "7980f00e-1872-47e6-83f0-3d231df70d77" (UID: "7980f00e-1872-47e6-83f0-3d231df70d77"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:14.393922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.393877 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podStartSLOduration=5.393858198 podStartE2EDuration="5.393858198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:11.415300799 +0000 UTC m=+1270.815856921" watchObservedRunningTime="2026-10-30 22:08:14.393858198 +0000 UTC m=+1273.794414320" Oct 30 22:08:14.492741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.492708 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lgg\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-kube-api-access-95lgg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.492741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.492735 5607 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7980f00e-1872-47e6-83f0-3d231df70d77-provider-credentials\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.938749 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk"] Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.939702 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" containerName="payload-processing" Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.939724 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" containerName="payload-processing" Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.939800 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" containerName="payload-processing" Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.939808 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" containerName="payload-processing" Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.939968 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" containerName="payload-processing" Oct 30 22:08:14.941342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.939987 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" containerName="payload-processing" Oct 30 22:08:14.945893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.945865 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:14.967471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.967444 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:08:14.968314 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:14.967939 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:08:15.010587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.009800 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d5e5df-openshift-default-dockercfg-vdt94\"" Oct 30 22:08:15.016983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.015354 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk"] Oct 30 22:08:15.040842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.040797 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.040859 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.040998 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.041040 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.041144 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.041179 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.041207 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwh94\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.041293 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.041422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.041384 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.142800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.142714 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.142800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.142754 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.142808 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.142835 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.142931 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143007 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143035 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwh94\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143108 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143159 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143201 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143428 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143476 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143559 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.143663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.143635 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.145410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.145389 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.146255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.146237 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.237156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.237076 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwh94\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.240778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.240708 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token\") pod \"e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.264547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.264518 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:15.269429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.269401 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59fbf69455-5km7v" Oct 30 22:08:15.270166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.270146 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6844c89498-74pp2" Oct 30 22:08:15.525548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.525519 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-74pp2"] Oct 30 22:08:15.528683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.528659 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6844c89498-74pp2"] Oct 30 22:08:15.639707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.639675 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59fbf69455-5km7v"] Oct 30 22:08:15.643826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.643802 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59fbf69455-5km7v"] Oct 30 22:08:15.723852 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.723825 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk"] Oct 30 22:08:15.726325 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:15.726294 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7d2999_08eb_4975_a1c9_d443d2313650.slice/crio-6e9b7bebd6ee55da4e7711a7ee2726712abe3418d63c674d8e643e54f29a8278 WatchSource:0}: Error finding container 6e9b7bebd6ee55da4e7711a7ee2726712abe3418d63c674d8e643e54f29a8278: Status 404 returned error can't find the container with id 6e9b7bebd6ee55da4e7711a7ee2726712abe3418d63c674d8e643e54f29a8278 Oct 30 22:08:15.728796 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.728761 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:15.728897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.728838 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:15.728897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.728884 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:15.966324 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.966289 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:08:15.966491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:15.966337 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:08:16.278301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.278248 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" event={"ID":"bb7d2999-08eb-4975-a1c9-d443d2313650","Type":"ContainerStarted","Data":"f64a9aecde9d48d4a138dd71dda4b8d2703cca558be64a7b61907cea8c4d1569"} Oct 30 22:08:16.278789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.278327 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" event={"ID":"bb7d2999-08eb-4975-a1c9-d443d2313650","Type":"ContainerStarted","Data":"6e9b7bebd6ee55da4e7711a7ee2726712abe3418d63c674d8e643e54f29a8278"} Oct 30 22:08:16.410002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.409922 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podStartSLOduration=2.409905603 podStartE2EDuration="2.409905603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:16.400923796 +0000 UTC m=+1275.801479918" watchObservedRunningTime="2026-10-30 22:08:16.409905603 +0000 UTC m=+1275.810461725" Oct 30 22:08:16.871180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.871146 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3c86cf-55b9-4ecc-ab77-654a56fb1a51" path="/var/lib/kubelet/pods/4a3c86cf-55b9-4ecc-ab77-654a56fb1a51/volumes" Oct 30 22:08:16.871628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.871607 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7980f00e-1872-47e6-83f0-3d231df70d77" path="/var/lib/kubelet/pods/7980f00e-1872-47e6-83f0-3d231df70d77/volumes" Oct 30 22:08:16.966614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.966584 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:08:16.966738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:16.966644 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:08:17.265345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.265309 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:17.270880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.270858 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:17.288356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.288331 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:17.289112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.289093 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:08:17.970973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.970931 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:17.971269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.971250 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:17.972046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:17.972027 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:08:19.052373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.052332 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f"] Oct 30 22:08:19.102877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.102848 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2529ef5-9d75f"] Oct 30 22:08:19.631990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.631944 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw"] Oct 30 22:08:19.636434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.636414 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.696827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.696794 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw"] Oct 30 22:08:19.742522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.742492 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:19.742675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.742545 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:08:19.801454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.801428 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhtg\" (UniqueName: \"kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.801604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.801554 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.801666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.801652 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.902995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.902890 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhtg\" (UniqueName: \"kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.902995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.902987 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.903175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.903075 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.903712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.903692 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.905435 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.905412 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:19.965486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:19.965454 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhtg\" (UniqueName: \"kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg\") pod \"maas-api-revoke-sub-4f424bc8-9d2lw\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:20.250706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:20.250677 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:20.626367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:20.626345 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw"] Oct 30 22:08:20.871661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:20.871596 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb899f2-bba8-460b-b35f-66741b879c16" path="/var/lib/kubelet/pods/0bb899f2-bba8-460b-b35f-66741b879c16/volumes" Oct 30 22:08:21.198523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.198454 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6"] Oct 30 22:08:21.202886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.202867 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.257355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.257333 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4cf21208\"" Oct 30 22:08:21.257355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.257346 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4cf21208-dockercfg-cl797\"" Oct 30 22:08:21.359009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.353727 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.359009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.354073 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpp8v\" (UniqueName: \"kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.359009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.358061 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6"] Oct 30 22:08:21.369045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.369004 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" event={"ID":"5e612835-bbfc-4ee4-aa6d-a5081df60854","Type":"ContainerStarted","Data":"3770323ebffff9362b09a562223fa8f5659dded1e8bb6a9dd21daae81ee9402e"} Oct 30 22:08:21.369260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.369071 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" event={"ID":"5e612835-bbfc-4ee4-aa6d-a5081df60854","Type":"ContainerStarted","Data":"b3f81e66a392f53789ddb2a95bd6f98fbf8f691ce6b63b2b0a80b56e1622a642"} Oct 30 22:08:21.389160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.389119 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm"] Oct 30 22:08:21.408838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.406011 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.410942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.410908 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm"] Oct 30 22:08:21.454990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.454889 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.455681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.455458 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpp8v\" (UniqueName: \"kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.455787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.455686 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.489866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.489823 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" podStartSLOduration=2.489811373 podStartE2EDuration="2.489811373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:21.487725578 +0000 UTC m=+1280.888281703" watchObservedRunningTime="2026-10-30 22:08:21.489811373 +0000 UTC m=+1280.890367519" Oct 30 22:08:21.523941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.523913 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpp8v\" (UniqueName: \"kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:21.556504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.556481 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.556701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.556676 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9t8\" (UniqueName: \"kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.657798 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.657770 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.657904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.657888 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9t8\" (UniqueName: \"kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.658459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.658438 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.720640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.720588 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9t8\" (UniqueName: \"kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.727453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.727434 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:21.816794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:21.816765 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:22.147006 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:22.144080 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe841774_2081_4101_8f1d_e188cef9d660.slice/crio-aaa359ab8a2e300b718620e7655fef3ef3c1e44bd6b8c47e5f47ea78d6375692 WatchSource:0}: Error finding container aaa359ab8a2e300b718620e7655fef3ef3c1e44bd6b8c47e5f47ea78d6375692: Status 404 returned error can't find the container with id aaa359ab8a2e300b718620e7655fef3ef3c1e44bd6b8c47e5f47ea78d6375692 Oct 30 22:08:22.148397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.148340 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm"] Oct 30 22:08:22.218988 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.218604 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6"] Oct 30 22:08:22.378531 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.378492 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" event={"ID":"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3","Type":"ContainerStarted","Data":"0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f"} Oct 30 22:08:22.379005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.378551 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" event={"ID":"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3","Type":"ContainerStarted","Data":"10bfb1787cb683221d38e582ee4ee41ca18796cab4e4f9b2966fe3b6f3c31b5f"} Oct 30 22:08:22.379005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.378614 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:22.380235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.380211 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" event={"ID":"be841774-2081-4101-8f1d-e188cef9d660","Type":"ContainerStarted","Data":"38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30"} Oct 30 22:08:22.380341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.380252 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" event={"ID":"be841774-2081-4101-8f1d-e188cef9d660","Type":"ContainerStarted","Data":"aaa359ab8a2e300b718620e7655fef3ef3c1e44bd6b8c47e5f47ea78d6375692"} Oct 30 22:08:22.380341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.380298 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:22.381797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.381776 5607 generic.go:356] "Generic (PLEG): container finished" podID="5e612835-bbfc-4ee4-aa6d-a5081df60854" containerID="3770323ebffff9362b09a562223fa8f5659dded1e8bb6a9dd21daae81ee9402e" exitCode=0 Oct 30 22:08:22.381917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.381823 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" event={"ID":"5e612835-bbfc-4ee4-aa6d-a5081df60854","Type":"ContainerDied","Data":"3770323ebffff9362b09a562223fa8f5659dded1e8bb6a9dd21daae81ee9402e"} Oct 30 22:08:22.498550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.498217 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" podStartSLOduration=1.49820022 podStartE2EDuration="1.49820022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:22.497555906 +0000 UTC m=+1281.898112027" watchObservedRunningTime="2026-10-30 22:08:22.49820022 +0000 UTC m=+1281.898756355" Oct 30 22:08:22.740795 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:22.740739 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" podStartSLOduration=1.740724793 podStartE2EDuration="1.740724793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:22.737494206 +0000 UTC m=+1282.138050327" watchObservedRunningTime="2026-10-30 22:08:22.740724793 +0000 UTC m=+1282.141280913" Oct 30 22:08:22.870405 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:08:22.870308 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:08:23.573907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.573878 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:23.581996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.581972 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca\") pod \"5e612835-bbfc-4ee4-aa6d-a5081df60854\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " Oct 30 22:08:23.582130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.582090 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg\" (UniqueName: \"kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg\") pod \"5e612835-bbfc-4ee4-aa6d-a5081df60854\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " Oct 30 22:08:23.582130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.582112 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca\") pod \"5e612835-bbfc-4ee4-aa6d-a5081df60854\" (UID: \"5e612835-bbfc-4ee4-aa6d-a5081df60854\") " Oct 30 22:08:23.582395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.582370 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca" pod "5e612835-bbfc-4ee4-aa6d-a5081df60854" (UID: "5e612835-bbfc-4ee4-aa6d-a5081df60854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:23.582558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.582543 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:23.584307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.584286 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca" pod "5e612835-bbfc-4ee4-aa6d-a5081df60854" (UID: "5e612835-bbfc-4ee4-aa6d-a5081df60854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:23.584408 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.584389 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg" pod "5e612835-bbfc-4ee4-aa6d-a5081df60854" (UID: "5e612835-bbfc-4ee4-aa6d-a5081df60854"). InnerVolumeSpecName "kube-api-access-tdhtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:23.683809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.683757 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdhtg\" (UniqueName: \"kubernetes.io/projected/5e612835-bbfc-4ee4-aa6d-a5081df60854-kube-api-access-tdhtg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:23.683809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:23.683781 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e612835-bbfc-4ee4-aa6d-a5081df60854-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:24.403107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:24.403081 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" Oct 30 22:08:24.403107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:24.403088 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw" event={"ID":"5e612835-bbfc-4ee4-aa6d-a5081df60854","Type":"ContainerDied","Data":"b3f81e66a392f53789ddb2a95bd6f98fbf8f691ce6b63b2b0a80b56e1622a642"} Oct 30 22:08:24.403365 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:24.403143 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3f81e66a392f53789ddb2a95bd6f98fbf8f691ce6b63b2b0a80b56e1622a642" Oct 30 22:08:25.053421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:25.053381 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr"] Oct 30 22:08:25.104434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:25.104404 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169d8e08-qtrqr"] Oct 30 22:08:26.336570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.336540 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm"] Oct 30 22:08:26.337301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.337282 5607 cpu_manager.go:414] "containerMap: removing container" podUID="5e612835-bbfc-4ee4-aa6d-a5081df60854" containerName="revoke-keys" Oct 30 22:08:26.337362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.337303 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e612835-bbfc-4ee4-aa6d-a5081df60854" containerName="revoke-keys" Oct 30 22:08:26.337444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.337434 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e612835-bbfc-4ee4-aa6d-a5081df60854" containerName="revoke-keys" Oct 30 22:08:26.341353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.341338 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.399614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.399587 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm"] Oct 30 22:08:26.413766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.413737 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcrx\" (UniqueName: \"kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx\") pod \"payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.413985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.413938 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.453978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.450258 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d5e5df\"" Oct 30 22:08:26.453978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.450551 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d5e5df-dockercfg-rk2cv\"" Oct 30 22:08:26.500350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.500298 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp"] Oct 30 22:08:26.538131 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.531503 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.542329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.533131 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.542517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.542497 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcrx\" (UniqueName: \"kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx\") pod \"payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.543118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.543082 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp"] Oct 30 22:08:26.543306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.543202 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.607357 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.607285 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcrx\" (UniqueName: \"kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx\") pod \"payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.644762 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.644723 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tnx\" (UniqueName: \"kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx\") pod \"payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.644908 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.644784 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.655407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.655379 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:26.746251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.746221 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tnx\" (UniqueName: \"kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx\") pod \"payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.746370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.746264 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.746931 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.746911 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.810751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.810724 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tnx\" (UniqueName: \"kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx\") pod \"payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.863285 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.863221 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:26.871598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:26.871571 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6051c04-fcce-4b4b-b0d6-8d10f3620985" path="/var/lib/kubelet/pods/d6051c04-fcce-4b4b-b0d6-8d10f3620985/volumes" Oct 30 22:08:27.035558 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:27.035528 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ba3daa_8dae_4413_b9f0_bdb34dd662ed.slice/crio-cdbffdbd12fc5220e368aa79eeecb51f748aa4a2491f691f01b7ef59870236f1 WatchSource:0}: Error finding container cdbffdbd12fc5220e368aa79eeecb51f748aa4a2491f691f01b7ef59870236f1: Status 404 returned error can't find the container with id cdbffdbd12fc5220e368aa79eeecb51f748aa4a2491f691f01b7ef59870236f1 Oct 30 22:08:27.040669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.040646 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm"] Oct 30 22:08:27.236379 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:27.236349 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd453d922_38f1_49ca_a480_20543fa744fa.slice/crio-47097b96cefe24f6795655ffb81aabe036c4b6f4cdc4f2bc3c3cec5fb2e6812a WatchSource:0}: Error finding container 47097b96cefe24f6795655ffb81aabe036c4b6f4cdc4f2bc3c3cec5fb2e6812a: Status 404 returned error can't find the container with id 47097b96cefe24f6795655ffb81aabe036c4b6f4cdc4f2bc3c3cec5fb2e6812a Oct 30 22:08:27.240250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.240227 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp"] Oct 30 22:08:27.435498 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.435451 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" event={"ID":"d453d922-38f1-49ca-a480-20543fa744fa","Type":"ContainerStarted","Data":"642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4"} Oct 30 22:08:27.436069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.435511 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" event={"ID":"d453d922-38f1-49ca-a480-20543fa744fa","Type":"ContainerStarted","Data":"47097b96cefe24f6795655ffb81aabe036c4b6f4cdc4f2bc3c3cec5fb2e6812a"} Oct 30 22:08:27.436069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.435612 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:27.437848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.437727 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" event={"ID":"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed","Type":"ContainerStarted","Data":"168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4"} Oct 30 22:08:27.437848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.437789 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" event={"ID":"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed","Type":"ContainerStarted","Data":"cdbffdbd12fc5220e368aa79eeecb51f748aa4a2491f691f01b7ef59870236f1"} Oct 30 22:08:27.437848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.437819 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:27.557743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.557632 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" podStartSLOduration=1.557615775 podStartE2EDuration="1.557615775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:27.553971214 +0000 UTC m=+1286.954527332" watchObservedRunningTime="2026-10-30 22:08:27.557615775 +0000 UTC m=+1286.958171897" Oct 30 22:08:27.676755 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:27.676684 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" podStartSLOduration=1.6766651110000002 podStartE2EDuration="1.676665111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:27.67191555 +0000 UTC m=+1287.072471672" watchObservedRunningTime="2026-10-30 22:08:27.676665111 +0000 UTC m=+1287.077221233" Oct 30 22:08:29.216978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.216176 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.677226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.677046 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.747152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.747092 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.792134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.792089 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.793413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.793365 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.865977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.865414 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:29.865977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.865464 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.896978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.882463 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:29.896978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.882513 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:29.981978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:29.980980 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.112490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.112411 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.220979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.218085 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.220979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.218729 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:08:30.622978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.612009 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt"] Oct 30 22:08:30.689978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.688131 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:30.756174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.756117 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.756842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.756822 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:08:30.784992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.783628 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:08:30.784992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.784297 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.834157 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.834593 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.840911 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.841232 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.845394 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.845621 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57nk5\" (UniqueName: \"kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:30.876044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:30.845667 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.039902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.039864 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57nk5\" (UniqueName: \"kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.047164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.047134 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.047518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.047496 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.126937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.126865 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.129336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.129310 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.156993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.156932 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt"] Oct 30 22:08:31.157326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.157292 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:08:31.157554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.157529 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:08:31.157724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.157710 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:08:31.181865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.181840 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:08:31.191144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.190669 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57nk5\" (UniqueName: \"kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5\") pod \"maas-api-revoke-sub-1c14b5fc-kk5nt\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.411023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.406219 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:31.841858 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:31.841455 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21596c24_589d_4d5e_9448_c219940f0342.slice/crio-319c106d98bac44baeaa47349b60cf7ae41ae51905449b77d6b370b28e53e61e WatchSource:0}: Error finding container 319c106d98bac44baeaa47349b60cf7ae41ae51905449b77d6b370b28e53e61e: Status 404 returned error can't find the container with id 319c106d98bac44baeaa47349b60cf7ae41ae51905449b77d6b370b28e53e61e Oct 30 22:08:31.853789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:31.853759 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt"] Oct 30 22:08:32.681806 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:32.681763 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" event={"ID":"21596c24-589d-4d5e-9448-c219940f0342","Type":"ContainerStarted","Data":"61312468e2bf49594c79bd5143620ded1a0a377a8532b570e9b4be858075276b"} Oct 30 22:08:32.682257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:32.681830 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" event={"ID":"21596c24-589d-4d5e-9448-c219940f0342","Type":"ContainerStarted","Data":"319c106d98bac44baeaa47349b60cf7ae41ae51905449b77d6b370b28e53e61e"} Oct 30 22:08:32.800163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:32.800122 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" podStartSLOduration=2.800109599 podStartE2EDuration="2.800109599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:32.799618495 +0000 UTC m=+1292.200174616" watchObservedRunningTime="2026-10-30 22:08:32.800109599 +0000 UTC m=+1292.200665719" Oct 30 22:08:33.393472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:33.393445 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:08:33.393748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:33.393510 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:08:33.691473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:33.691440 5607 generic.go:356] "Generic (PLEG): container finished" podID="21596c24-589d-4d5e-9448-c219940f0342" containerID="61312468e2bf49594c79bd5143620ded1a0a377a8532b570e9b4be858075276b" exitCode=0 Oct 30 22:08:33.691865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:33.691522 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" event={"ID":"21596c24-589d-4d5e-9448-c219940f0342","Type":"ContainerDied","Data":"61312468e2bf49594c79bd5143620ded1a0a377a8532b570e9b4be858075276b"} Oct 30 22:08:34.861616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.861592 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:34.943477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.943453 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca\") pod \"21596c24-589d-4d5e-9448-c219940f0342\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " Oct 30 22:08:34.943620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.943511 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca\") pod \"21596c24-589d-4d5e-9448-c219940f0342\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " Oct 30 22:08:34.943620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.943578 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5\" (UniqueName: \"kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5\") pod \"21596c24-589d-4d5e-9448-c219940f0342\" (UID: \"21596c24-589d-4d5e-9448-c219940f0342\") " Oct 30 22:08:34.943862 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.943838 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca" pod "21596c24-589d-4d5e-9448-c219940f0342" (UID: "21596c24-589d-4d5e-9448-c219940f0342"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:34.944125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.944104 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21596c24-589d-4d5e-9448-c219940f0342-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.945683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.945659 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca" pod "21596c24-589d-4d5e-9448-c219940f0342" (UID: "21596c24-589d-4d5e-9448-c219940f0342"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:34.945683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:34.945678 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5" pod "21596c24-589d-4d5e-9448-c219940f0342" (UID: "21596c24-589d-4d5e-9448-c219940f0342"). InnerVolumeSpecName "kube-api-access-57nk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:35.044888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:35.044856 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21596c24-589d-4d5e-9448-c219940f0342-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.044888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:35.044884 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57nk5\" (UniqueName: \"kubernetes.io/projected/21596c24-589d-4d5e-9448-c219940f0342-kube-api-access-57nk5\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.706678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:35.706648 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" Oct 30 22:08:35.706856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:35.706684 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt" event={"ID":"21596c24-589d-4d5e-9448-c219940f0342","Type":"ContainerDied","Data":"319c106d98bac44baeaa47349b60cf7ae41ae51905449b77d6b370b28e53e61e"} Oct 30 22:08:35.706856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:35.706730 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="319c106d98bac44baeaa47349b60cf7ae41ae51905449b77d6b370b28e53e61e" Oct 30 22:08:36.052479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:36.052407 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj"] Oct 30 22:08:36.056799 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:36.056768 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfad996-lckxj"] Oct 30 22:08:36.871610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:36.871569 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7c8905-7410-4af6-b7eb-192b2e2c071e" path="/var/lib/kubelet/pods/6b7c8905-7410-4af6-b7eb-192b2e2c071e/volumes" Oct 30 22:08:38.465851 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:38.465824 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:08:38.466280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:38.465888 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:08:45.052733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:45.052702 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh"] Oct 30 22:08:45.105708 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:45.105675 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adba886-7l2hh"] Oct 30 22:08:46.870351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:46.870322 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f779b0f-ffc7-482f-ab07-d7348e6b8c54" path="/var/lib/kubelet/pods/7f779b0f-ffc7-482f-ab07-d7348e6b8c54/volumes" Oct 30 22:08:46.986303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:46.986275 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md"] Oct 30 22:08:46.986897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:46.986884 5607 cpu_manager.go:414] "containerMap: removing container" podUID="21596c24-589d-4d5e-9448-c219940f0342" containerName="revoke-keys" Oct 30 22:08:46.986939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:46.986899 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21596c24-589d-4d5e-9448-c219940f0342" containerName="revoke-keys" Oct 30 22:08:46.987021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:46.987011 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="21596c24-589d-4d5e-9448-c219940f0342" containerName="revoke-keys" Oct 30 22:08:46.992017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:46.992001 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.096587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.096560 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4cf21208-serving-cert\"" Oct 30 22:08:47.096587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.096596 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwpgv\"" Oct 30 22:08:47.096769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.096670 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:08:47.100493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.100473 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md"] Oct 30 22:08:47.178006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.177877 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.178199 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.178072 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkksp\" (UniqueName: \"kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.178199 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.178121 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.178337 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.178210 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.279740 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.279700 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkksp\" (UniqueName: \"kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.279945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.279755 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.279945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.279809 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.279945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.279861 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.279945 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:08:47.279874 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-serving-cert: secret "maas-api-e2e-mt-4cf21208-serving-cert" not found Oct 30 22:08:47.280222 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:08:47.279983 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls podName:a391201c-a4d6-454b-b829-060b7dc6d5c9 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:47.779943094 +0000 UTC m=+1307.180499215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls") pod "maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" (UID: "a391201c-a4d6-454b-b829-060b7dc6d5c9") : secret "maas-api-e2e-mt-4cf21208-serving-cert" not found Oct 30 22:08:47.280643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.280624 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.282423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.282403 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.342671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.342640 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkksp\" (UniqueName: \"kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.786702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.786668 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.789405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.789378 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\") pod \"maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:47.905696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:47.905669 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:48.280704 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:48.280673 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda391201c_a4d6_454b_b829_060b7dc6d5c9.slice/crio-e6308f4f83db74ebd46face7b36c4123f6a4b720a936e771a364270da2d5730d WatchSource:0}: Error finding container e6308f4f83db74ebd46face7b36c4123f6a4b720a936e771a364270da2d5730d: Status 404 returned error can't find the container with id e6308f4f83db74ebd46face7b36c4123f6a4b720a936e771a364270da2d5730d Oct 30 22:08:48.282378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:48.282353 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:08:48.284688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:48.284664 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md"] Oct 30 22:08:48.803419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:48.803381 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" event={"ID":"a391201c-a4d6-454b-b829-060b7dc6d5c9","Type":"ContainerStarted","Data":"e6308f4f83db74ebd46face7b36c4123f6a4b720a936e771a364270da2d5730d"} Oct 30 22:08:50.820660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:50.820623 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" event={"ID":"a391201c-a4d6-454b-b829-060b7dc6d5c9","Type":"ContainerStarted","Data":"94a1a021fdbbddb7e044adff367f4e7fafac387bfeaa6d98af391d5a3e340812"} Oct 30 22:08:50.821030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:50.820738 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:50.949438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:50.949388 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" podStartSLOduration=2.520803028 podStartE2EDuration="4.949375301s" totalImagesPullingTime="2.428572273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:50.945488361 +0000 UTC m=+1310.346044479" watchObservedRunningTime="2026-10-30 22:08:50.949375301 +0000 UTC m=+1310.349931422" Oct 30 22:08:51.905978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:51.905673 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r"] Oct 30 22:08:51.934328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:51.934300 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.048084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.048048 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d5e5df-serving-cert\"" Oct 30 22:08:52.051976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.051928 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r"] Oct 30 22:08:52.096978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.074704 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpnt\" (UniqueName: \"kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.096978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.074813 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.096978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.074873 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.096978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.074989 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.268038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.176168 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpnt\" (UniqueName: \"kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.286978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.268426 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.287252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.287227 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.287431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.287414 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.290049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.269282 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.309096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.309057 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.314971 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.314537 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.375896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.375820 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpnt\" (UniqueName: \"kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt\") pod \"maas-api-e2e-api-a-d5e5df-d555b68-57n7r\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:52.400414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:52.400138 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:53.305643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:53.244035 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 22:08:53.342091 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:08:53.325900 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:08:53.342091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:53.327583 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:53.342523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:53.342417 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:53.342523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:53.342467 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:53.361047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:53.342758 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:53.786130 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:53.786090 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d258df_aeb2_48ab_98f1_1caf1a50b065.slice/crio-0f212ea5bea21dd02b051e0a7427bd62fee4e2695a1d59e0ed3d049dd0832657 WatchSource:0}: Error finding container 0f212ea5bea21dd02b051e0a7427bd62fee4e2695a1d59e0ed3d049dd0832657: Status 404 returned error can't find the container with id 0f212ea5bea21dd02b051e0a7427bd62fee4e2695a1d59e0ed3d049dd0832657 Oct 30 22:08:53.786713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:53.786672 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r"] Oct 30 22:08:54.334494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.334455 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" event={"ID":"b9d258df-aeb2-48ab-98f1-1caf1a50b065","Type":"ContainerStarted","Data":"0f212ea5bea21dd02b051e0a7427bd62fee4e2695a1d59e0ed3d049dd0832657"} Oct 30 22:08:54.746016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.745987 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp"] Oct 30 22:08:54.751578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.751559 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.847694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.847666 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.847871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.847732 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.847871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.847780 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.848009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.847864 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.848009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.847922 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.848086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.848045 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.848086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.848068 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.848225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.848112 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrsdd\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.848364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.848340 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949285 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949411 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949460 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949504 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949534 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949563 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949651 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.949717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949677 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.950291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.949749 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrsdd\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.950866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.950819 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.954900 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.954718 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.956475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.955514 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.956475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.956431 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.956893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.956870 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.977979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.965185 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp"] Oct 30 22:08:54.990020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.979469 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:54.990505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:54.990479 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:55.015912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.015877 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:55.016072 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.016053 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrsdd\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:55.065555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.065526 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:55.344010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.343912 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" event={"ID":"b9d258df-aeb2-48ab-98f1-1caf1a50b065","Type":"ContainerStarted","Data":"05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9"} Oct 30 22:08:55.344010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.344000 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:08:55.465803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.465758 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" podStartSLOduration=3.568758416 podStartE2EDuration="4.465745352s" totalImagesPullingTime="896.986936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:55.462622542 +0000 UTC m=+1314.863178663" watchObservedRunningTime="2026-10-30 22:08:55.465745352 +0000 UTC m=+1314.866301473" Oct 30 22:08:55.491019 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:08:55.490996 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0dc286_ed78_40f8_8f03_99132ba5aa74.slice/crio-b97cb11c0988357ba8df5fb3fed41b5028ec3f5c4f89252fbf39f0ff3c3fdfb4 WatchSource:0}: Error finding container b97cb11c0988357ba8df5fb3fed41b5028ec3f5c4f89252fbf39f0ff3c3fdfb4: Status 404 returned error can't find the container with id b97cb11c0988357ba8df5fb3fed41b5028ec3f5c4f89252fbf39f0ff3c3fdfb4 Oct 30 22:08:55.493160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.493132 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:55.493246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.493192 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:55.493246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.493231 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:55.518215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:55.518191 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp"] Oct 30 22:08:56.353599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:56.353564 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" event={"ID":"3b0dc286-ed78-40f8-8f03-99132ba5aa74","Type":"ContainerStarted","Data":"fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675"} Oct 30 22:08:56.354052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:56.353616 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" event={"ID":"3b0dc286-ed78-40f8-8f03-99132ba5aa74","Type":"ContainerStarted","Data":"b97cb11c0988357ba8df5fb3fed41b5028ec3f5c4f89252fbf39f0ff3c3fdfb4"} Oct 30 22:08:56.481053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:56.480926 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podStartSLOduration=2.48091072 podStartE2EDuration="2.48091072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:56.477352046 +0000 UTC m=+1315.877908161" watchObservedRunningTime="2026-10-30 22:08:56.48091072 +0000 UTC m=+1315.881466841" Oct 30 22:08:56.928090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:56.928058 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:08:57.066330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:57.066298 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:08:57.067565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:57.067538 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:08:57.067712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:57.067587 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:08:58.066615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:58.066581 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:08:58.067009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:58.066639 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:08:59.066248 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:59.066219 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:08:59.066416 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:08:59.066269 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:09:00.066433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:00.066402 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:09:00.066825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:00.066457 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:09:01.066292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:01.066260 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:09:01.066445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:01.066320 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:09:01.154834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:01.154808 5607 scope.go:122] "RemoveContainer" containerID="e3221fdc0bd06b7038bb3a383bb68fc4b824da42b2741f8cfaf4aadc1193d935" Oct 30 22:09:01.164828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:01.164802 5607 scope.go:122] "RemoveContainer" containerID="38654876f2d1cc04ffa2d0b33571197491bc5d70dbcd4d59e7af0a038f5f5379" Oct 30 22:09:01.176101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:01.176083 5607 scope.go:122] "RemoveContainer" containerID="9aca96536fe814afb96d3bf37231c026448549eaf0146714ce131b47c86bc1af" Oct 30 22:09:01.358931 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:01.358866 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:09:02.066547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:02.066503 5607 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:09:02.066744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:02.066557 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 22:09:03.070317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:03.070286 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:09:03.070769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:03.070752 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:09:03.071354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:03.071340 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:09:04.173436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:04.173406 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm"] Oct 30 22:09:04.174044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:04.173643 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" containerID="cri-o://c9904186f9167e07d65605b921c1b43025acdea9059b8f1f8b3da2b63396b1e7" gracePeriod=30 Oct 30 22:09:09.454112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.454055 5607 generic.go:356] "Generic (PLEG): container finished" podID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerID="c9904186f9167e07d65605b921c1b43025acdea9059b8f1f8b3da2b63396b1e7" exitCode=0 Oct 30 22:09:09.454433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.454139 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" event={"ID":"1511ebe9-bdff-44e5-a614-9e70273e7e33","Type":"ContainerDied","Data":"c9904186f9167e07d65605b921c1b43025acdea9059b8f1f8b3da2b63396b1e7"} Oct 30 22:09:09.454433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.454155 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 22:09:09.454433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.454225 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" event={"ID":"1511ebe9-bdff-44e5-a614-9e70273e7e33","Type":"ContainerDied","Data":"4b87e026145677642814ae87e227d9336a912801429550483aa9916381d0d771"} Oct 30 22:09:09.454433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.454248 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b87e026145677642814ae87e227d9336a912801429550483aa9916381d0d771" Oct 30 22:09:09.454433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.454261 5607 scope.go:122] "RemoveContainer" containerID="e0dc21d5b3a684ef53948b055fadc035aba7a8b727687531262a4b826fbb82b4" Oct 30 22:09:09.550045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550024 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550100 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550131 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550212 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550268 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550306 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:09.550424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550326 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550377 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550407 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550444 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo\") pod \"1511ebe9-bdff-44e5-a614-9e70273e7e33\" (UID: \"1511ebe9-bdff-44e5-a614-9e70273e7e33\") " Oct 30 22:09:09.550632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550505 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:09.550632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550520 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:09.550792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550735 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:09.550842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.550784 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:09.551239 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.551211 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.551596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.551244 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1511ebe9-bdff-44e5-a614-9e70273e7e33-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.551596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.551261 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.551596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.551276 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.551596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.551290 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.552893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.552872 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:09.553292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.553264 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "kube-api-access-xvsmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:09.554025 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.554003 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:09.554025 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.554017 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy" pod "1511ebe9-bdff-44e5-a614-9e70273e7e33" (UID: "1511ebe9-bdff-44e5-a614-9e70273e7e33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:09.652520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.652495 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.652520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.652519 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.652682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.652529 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvsmp\" (UniqueName: \"kubernetes.io/projected/1511ebe9-bdff-44e5-a614-9e70273e7e33-kube-api-access-xvsmp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:09.652682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:09.652540 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1511ebe9-bdff-44e5-a614-9e70273e7e33-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:10.462762 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:10.462724 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm" Oct 30 22:09:10.708472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:10.708439 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm"] Oct 30 22:09:10.715381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:10.715318 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-ptdwm"] Oct 30 22:09:10.871552 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:10.871520 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" path="/var/lib/kubelet/pods/1511ebe9-bdff-44e5-a614-9e70273e7e33/volumes" Oct 30 22:09:22.865061 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:09:22.865024 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:09:25.615871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:25.615838 5607 generic.go:356] "Generic (PLEG): container finished" podID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerID="f64a9aecde9d48d4a138dd71dda4b8d2703cca558be64a7b61907cea8c4d1569" exitCode=0 Oct 30 22:09:25.616352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:25.615910 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" event={"ID":"bb7d2999-08eb-4975-a1c9-d443d2313650","Type":"ContainerDied","Data":"f64a9aecde9d48d4a138dd71dda4b8d2703cca558be64a7b61907cea8c4d1569"} Oct 30 22:09:25.616352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:25.616275 5607 scope.go:122] "RemoveContainer" containerID="f64a9aecde9d48d4a138dd71dda4b8d2703cca558be64a7b61907cea8c4d1569" Oct 30 22:09:25.617571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:25.617541 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:25.617700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:25.617627 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:25.617700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:25.617680 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:26.637011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:26.630201 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" event={"ID":"bb7d2999-08eb-4975-a1c9-d443d2313650","Type":"ContainerStarted","Data":"1fbbda21afe90d34824b3f603359f80930dc8fc468b330d95333c93cba779d9e"} Oct 30 22:09:27.264773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:27.264742 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:09:27.264947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:27.264910 5607 patch_prober.go:29] interesting pod/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:09:27.265016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:27.264988 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:09:28.265705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:28.265673 5607 patch_prober.go:29] interesting pod/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:09:28.266122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:28.265733 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:09:29.265742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:29.265707 5607 patch_prober.go:29] interesting pod/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:09:29.266159 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:29.265771 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:09:30.265713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:30.265671 5607 patch_prober.go:29] interesting pod/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:09:30.265887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:30.265740 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:09:31.265225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:31.265187 5607 patch_prober.go:29] interesting pod/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:09:31.265401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:31.265244 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:09:32.269206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:32.269179 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:09:32.269723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:32.269702 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:09:32.270429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:32.270410 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:09:36.931120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.931089 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5"] Oct 30 22:09:36.931982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.931927 5607 cpu_manager.go:414] "containerMap: removing container" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" Oct 30 22:09:36.931982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.931969 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" Oct 30 22:09:36.932149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.931991 5607 cpu_manager.go:414] "containerMap: removing container" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" Oct 30 22:09:36.932149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.932000 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" Oct 30 22:09:36.932252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.932228 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" Oct 30 22:09:36.932252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.932244 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="1511ebe9-bdff-44e5-a614-9e70273e7e33" containerName="istio-proxy" Oct 30 22:09:36.936939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.936919 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:36.992122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:36.992087 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dbe0a7-openshift-default-dockercfg-l6j9x\"" Oct 30 22:09:37.059653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.055316 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060392 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060286 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060607 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5"] Oct 30 22:09:37.060748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060638 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060686 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060728 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060783 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060813 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.060918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060837 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnn49\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.061110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.060922 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162594 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162725 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162765 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162793 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162827 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162881 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162908 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.162976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.162932 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnn49\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.163110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.163040 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.163421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.163394 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.163535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.163426 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.163691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.163672 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.163834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.163677 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.163938 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.163848 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.165900 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.165875 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.166046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.166028 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.227877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.227840 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.228124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.228099 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnn49\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49\") pod \"e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.252140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.252112 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:37.690236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.690205 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5"] Oct 30 22:09:37.719269 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:09:37.719243 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40058b19_0e7f_4b5d_a417_e296ca5e993d.slice/crio-bc594a929cc1d176d434bda2517199cf3ac804d900da4e96024f045451671a75 WatchSource:0}: Error finding container bc594a929cc1d176d434bda2517199cf3ac804d900da4e96024f045451671a75: Status 404 returned error can't find the container with id bc594a929cc1d176d434bda2517199cf3ac804d900da4e96024f045451671a75 Oct 30 22:09:37.721561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.721524 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:37.721688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.721607 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:37.721688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:37.721648 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:38.725897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:38.725860 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" event={"ID":"40058b19-0e7f-4b5d-a417-e296ca5e993d","Type":"ContainerStarted","Data":"6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a"} Oct 30 22:09:38.726291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:38.725942 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" event={"ID":"40058b19-0e7f-4b5d-a417-e296ca5e993d","Type":"ContainerStarted","Data":"bc594a929cc1d176d434bda2517199cf3ac804d900da4e96024f045451671a75"} Oct 30 22:09:38.904110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:38.904064 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podStartSLOduration=2.9040505039999998 podStartE2EDuration="2.904050504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:38.899680633 +0000 UTC m=+1358.300236766" watchObservedRunningTime="2026-10-30 22:09:38.904050504 +0000 UTC m=+1358.304606624" Oct 30 22:09:39.252826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:39.252786 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:39.258903 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:39.258879 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:39.733283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:39.733258 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:39.734079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:39.734061 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:09:45.361875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.361838 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl"] Oct 30 22:09:45.366412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.366389 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.423967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.423927 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl"] Oct 30 22:09:45.475309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.475194 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dbe0a7-dockercfg-pzqwb\"" Oct 30 22:09:45.475309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.475282 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dbe0a7\"" Oct 30 22:09:45.530493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.530456 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2"] Oct 30 22:09:45.558261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.558234 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.566984 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.566230 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2"] Oct 30 22:09:45.598023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.597992 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.598199 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.598037 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.598266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.598205 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rq2\" (UniqueName: \"kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2\") pod \"payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.598374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.598357 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6vh\" (UniqueName: \"kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh\") pod \"payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.700013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.699989 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6vh\" (UniqueName: \"kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh\") pod \"payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.700161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.700093 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.700161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.700113 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.700241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.700162 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rq2\" (UniqueName: \"kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2\") pod \"payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.700842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.700820 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.700878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.700837 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.762325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.762299 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6vh\" (UniqueName: \"kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh\") pod \"payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:45.762440 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.762327 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rq2\" (UniqueName: \"kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2\") pod \"payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.901112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.901085 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:45.981264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:45.981233 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:46.274817 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:09:46.274786 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b74249f_5fd2_4d43_aabe_89200c159d60.slice/crio-8bc23d9db40fdad8175903ee28973fc537054374e504373eaba005f055655967 WatchSource:0}: Error finding container 8bc23d9db40fdad8175903ee28973fc537054374e504373eaba005f055655967: Status 404 returned error can't find the container with id 8bc23d9db40fdad8175903ee28973fc537054374e504373eaba005f055655967 Oct 30 22:09:46.278930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.278907 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2"] Oct 30 22:09:46.380157 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:09:46.380128 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65fa155_efe9_4f4e_b149_101ec99272f9.slice/crio-f62c64e4a28966fcc345159f211157870ac1117888fe9d73888feb38a93202e9 WatchSource:0}: Error finding container f62c64e4a28966fcc345159f211157870ac1117888fe9d73888feb38a93202e9: Status 404 returned error can't find the container with id f62c64e4a28966fcc345159f211157870ac1117888fe9d73888feb38a93202e9 Oct 30 22:09:46.385324 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.385238 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl"] Oct 30 22:09:46.785827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.785785 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" event={"ID":"a65fa155-efe9-4f4e-b149-101ec99272f9","Type":"ContainerStarted","Data":"262df72b2da409843bdf8ff98beec0a348c215bb629b52b5c3bdd896910565d6"} Oct 30 22:09:46.786035 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.785879 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:46.786035 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.785905 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" event={"ID":"a65fa155-efe9-4f4e-b149-101ec99272f9","Type":"ContainerStarted","Data":"f62c64e4a28966fcc345159f211157870ac1117888fe9d73888feb38a93202e9"} Oct 30 22:09:46.787555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.787531 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" event={"ID":"8b74249f-5fd2-4d43-aabe-89200c159d60","Type":"ContainerStarted","Data":"5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0"} Oct 30 22:09:46.787684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.787572 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" event={"ID":"8b74249f-5fd2-4d43-aabe-89200c159d60","Type":"ContainerStarted","Data":"8bc23d9db40fdad8175903ee28973fc537054374e504373eaba005f055655967"} Oct 30 22:09:46.787684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.787626 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:46.906546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:46.906484 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" podStartSLOduration=1.906466703 podStartE2EDuration="1.906466703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:46.902761886 +0000 UTC m=+1366.303318007" watchObservedRunningTime="2026-10-30 22:09:46.906466703 +0000 UTC m=+1366.307022823" Oct 30 22:09:47.026902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:47.025368 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" podStartSLOduration=2.025352341 podStartE2EDuration="2.025352341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:47.022203303 +0000 UTC m=+1366.422759453" watchObservedRunningTime="2026-10-30 22:09:47.025352341 +0000 UTC m=+1366.425908477" Oct 30 22:09:49.527192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.527161 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd"] Oct 30 22:09:49.531498 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.531480 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.585786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.585764 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b47e8d-openshift-default-dockercfg-bxx6t\"" Oct 30 22:09:49.641450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.641427 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd"] Oct 30 22:09:49.650197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650166 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650227 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650300 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650356 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650383 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650406 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpm6\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650513 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650616 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.650737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.650687 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.751812 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.751775 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.751820 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.751861 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.751893 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.751924 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpm6\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752001 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752278 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752125 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752278 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752230 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752270 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752291 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752293 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752560 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.752907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752795 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.753015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.752997 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.754430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.754411 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.754713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.754695 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.820257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.820199 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpm6\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.820486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.820469 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:49.845743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:49.845688 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:50.271008 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:09:50.270976 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212ef815_88f1_42d0_820d_d20ff273e1a6.slice/crio-a83030af7eda54e2f4733be301860a125809a72edf368b337bff430d4de56632 WatchSource:0}: Error finding container a83030af7eda54e2f4733be301860a125809a72edf368b337bff430d4de56632: Status 404 returned error can't find the container with id a83030af7eda54e2f4733be301860a125809a72edf368b337bff430d4de56632 Oct 30 22:09:50.273271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.273238 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:50.273360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.273310 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:50.273360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.273341 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:50.274742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.274716 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd"] Oct 30 22:09:50.826634 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.826594 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" event={"ID":"212ef815-88f1-42d0-820d-d20ff273e1a6","Type":"ContainerStarted","Data":"01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400"} Oct 30 22:09:50.827032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.826655 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" event={"ID":"212ef815-88f1-42d0-820d-d20ff273e1a6","Type":"ContainerStarted","Data":"a83030af7eda54e2f4733be301860a125809a72edf368b337bff430d4de56632"} Oct 30 22:09:50.846181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.846157 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:50.955074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:50.955003 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podStartSLOduration=1.954983086 podStartE2EDuration="1.954983086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:50.950725114 +0000 UTC m=+1370.351281235" watchObservedRunningTime="2026-10-30 22:09:50.954983086 +0000 UTC m=+1370.355539208" Oct 30 22:09:51.847618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:51.847587 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:09:51.848128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:51.847641 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:09:52.846651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:52.846620 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:09:52.846819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:52.846667 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:09:52.865466 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:09:52.865429 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:09:53.846611 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:53.846574 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:09:53.846777 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:53.846632 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:09:54.847006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:54.846941 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:09:54.847385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:54.847023 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:09:55.846403 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:55.846369 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:09:55.846574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:55.846427 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:09:56.847011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:56.846938 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:09:56.847396 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:56.847041 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:09:57.798603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:57.798577 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:09:57.798774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:57.798640 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:09:58.520701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:58.520660 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:58.521254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:58.521224 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:09:58.522325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:09:58.522300 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:10:01.239898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:01.239865 5607 scope.go:122] "RemoveContainer" containerID="e07235c353950345ffaa50c02d31756de6db68a55fcd04dc14e83372a28b4e41" Oct 30 22:10:01.250560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:01.250538 5607 scope.go:122] "RemoveContainer" containerID="c9904186f9167e07d65605b921c1b43025acdea9059b8f1f8b3da2b63396b1e7" Oct 30 22:10:01.259837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:01.259820 5607 scope.go:122] "RemoveContainer" containerID="ff63e36944e7177404307e9ff9a6105e8648874b6415998dc74674ebe698d2af" Oct 30 22:10:01.268584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:01.268556 5607 scope.go:122] "RemoveContainer" containerID="eea665249166e5eb9bec27e7a169341f0d9476c232a2fa51405a749d151b09bb" Oct 30 22:10:01.279887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:01.279864 5607 scope.go:122] "RemoveContainer" containerID="24c0f579ff658a795f15a6fe688372b31834d30681bd6c62d17e8a73367e223d" Oct 30 22:10:18.708220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.708186 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv"] Oct 30 22:10:18.712747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.712728 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:18.817483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.817442 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-872730\"" Oct 30 22:10:18.817643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.817453 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-872730-dockercfg-nccl8\"" Oct 30 22:10:18.821654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.821631 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv"] Oct 30 22:10:18.913129 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.905429 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zvk\" (UniqueName: \"kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk\") pod \"payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:18.913129 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.905537 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:18.942986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.942043 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr"] Oct 30 22:10:18.949974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.947897 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr"] Oct 30 22:10:18.949974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:18.948048 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.006839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.006759 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zvk\" (UniqueName: \"kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk\") pod \"payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:19.006839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.006833 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:19.007079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.006859 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume\") pod \"payload-processing-e2e-api-a-872730-5874968845-brksr\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.007079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.006978 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tv46\" (UniqueName: \"kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46\") pod \"payload-processing-e2e-api-a-872730-5874968845-brksr\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.007574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.007554 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:19.072175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.072145 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zvk\" (UniqueName: \"kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk\") pod \"payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:19.107868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.107839 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume\") pod \"payload-processing-e2e-api-a-872730-5874968845-brksr\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.108041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.108007 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tv46\" (UniqueName: \"kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46\") pod \"payload-processing-e2e-api-a-872730-5874968845-brksr\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.108542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.108524 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume\") pod \"payload-processing-e2e-api-a-872730-5874968845-brksr\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.170844 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.170823 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tv46\" (UniqueName: \"kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46\") pod \"payload-processing-e2e-api-a-872730-5874968845-brksr\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.267947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.267892 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:19.327227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.327192 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:19.641548 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:10:19.641511 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252cf1fd_2366_4e63_bcd0_e1af768fec53.slice/crio-1e2e38778085a85ee8af42adc827121ed16f63f10c3b2a7479f0cfd728a48314 WatchSource:0}: Error finding container 1e2e38778085a85ee8af42adc827121ed16f63f10c3b2a7479f0cfd728a48314: Status 404 returned error can't find the container with id 1e2e38778085a85ee8af42adc827121ed16f63f10c3b2a7479f0cfd728a48314 Oct 30 22:10:19.645440 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.645415 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr"] Oct 30 22:10:19.737419 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:10:19.737394 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba75ef_f1f3_44c0_b296_8668a1c514af.slice/crio-b03957fb43d5cc288009ad95fc52c9c1739a4ac60b55994d02b600dc646e53c0 WatchSource:0}: Error finding container b03957fb43d5cc288009ad95fc52c9c1739a4ac60b55994d02b600dc646e53c0: Status 404 returned error can't find the container with id b03957fb43d5cc288009ad95fc52c9c1739a4ac60b55994d02b600dc646e53c0 Oct 30 22:10:19.740375 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:19.740347 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv"] Oct 30 22:10:20.053512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.053475 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" event={"ID":"252cf1fd-2366-4e63-bcd0-e1af768fec53","Type":"ContainerStarted","Data":"f82de597e0f7abb502673b969def000f9ba7b5d50849e6c8a8f76b723f1e118b"} Oct 30 22:10:20.053702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.053542 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" event={"ID":"252cf1fd-2366-4e63-bcd0-e1af768fec53","Type":"ContainerStarted","Data":"1e2e38778085a85ee8af42adc827121ed16f63f10c3b2a7479f0cfd728a48314"} Oct 30 22:10:20.053844 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.053817 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:20.055406 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.055382 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" event={"ID":"66ba75ef-f1f3-44c0-b296-8668a1c514af","Type":"ContainerStarted","Data":"d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981"} Oct 30 22:10:20.055517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.055426 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" event={"ID":"66ba75ef-f1f3-44c0-b296-8668a1c514af","Type":"ContainerStarted","Data":"b03957fb43d5cc288009ad95fc52c9c1739a4ac60b55994d02b600dc646e53c0"} Oct 30 22:10:20.055517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.055473 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:20.176286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.176240 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" podStartSLOduration=2.176227039 podStartE2EDuration="2.176227039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:20.174559228 +0000 UTC m=+1399.575115349" watchObservedRunningTime="2026-10-30 22:10:20.176227039 +0000 UTC m=+1399.576783159" Oct 30 22:10:20.297838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:20.297775 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" podStartSLOduration=2.297756264 podStartE2EDuration="2.297756264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:20.293184932 +0000 UTC m=+1399.693741052" watchObservedRunningTime="2026-10-30 22:10:20.297756264 +0000 UTC m=+1399.698312387" Oct 30 22:10:22.863896 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:10:22.863861 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:10:31.065725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:31.065693 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:10:31.066134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:31.065750 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:10:52.866182 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:10:52.866133 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:10:55.207163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.205890 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": read tcp 10.133.0.2:57380->10.133.0.118:15021: read: connection reset by peer" start-of-body= Oct 30 22:10:55.207163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.205997 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": read tcp 10.133.0.2:57380->10.133.0.118:15021: read: connection reset by peer" Oct 30 22:10:55.367171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.367112 5607 generic.go:356] "Generic (PLEG): container finished" podID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerID="6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a" exitCode=0 Oct 30 22:10:55.367281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.367171 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" event={"ID":"40058b19-0e7f-4b5d-a417-e296ca5e993d","Type":"ContainerDied","Data":"6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a"} Oct 30 22:10:55.367564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.367546 5607 scope.go:122] "RemoveContainer" containerID="6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a" Oct 30 22:10:55.369021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.368995 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:55.369088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.369068 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:55.369137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:55.369124 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:56.102977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.102004 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd"] Oct 30 22:10:56.127977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.127518 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8"] Oct 30 22:10:56.127977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.127914 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.134898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.134371 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.183537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.183457 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-161111\"" Oct 30 22:10:56.183775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.183673 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-161111-dockercfg-p2jxf\"" Oct 30 22:10:56.197142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.197095 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume\") pod \"payload-processing-e2e-api-a-161111-786d9b8465-52bs8\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.206053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.206009 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl4hq\" (UniqueName: \"kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq\") pod \"payload-processing-e2e-api-a-161111-786d9b8465-52bs8\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.206479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.206456 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.206803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.206781 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rl4\" (UniqueName: \"kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4\") pod \"payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.306980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.301019 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd"] Oct 30 22:10:56.328327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.328278 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl4hq\" (UniqueName: \"kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq\") pod \"payload-processing-e2e-api-a-161111-786d9b8465-52bs8\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.328550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.328458 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.328639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.328611 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rl4\" (UniqueName: \"kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4\") pod \"payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.328719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.328696 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume\") pod \"payload-processing-e2e-api-a-161111-786d9b8465-52bs8\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.355072 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.339042 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.361030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.359483 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume\") pod \"payload-processing-e2e-api-a-161111-786d9b8465-52bs8\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.362602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.362527 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8"] Oct 30 22:10:56.388787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.388747 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" event={"ID":"40058b19-0e7f-4b5d-a417-e296ca5e993d","Type":"ContainerStarted","Data":"8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee"} Oct 30 22:10:56.434880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.434845 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rl4\" (UniqueName: \"kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4\") pod \"payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.437680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.437654 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl4hq\" (UniqueName: \"kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq\") pod \"payload-processing-e2e-api-a-161111-786d9b8465-52bs8\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.452133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.452111 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:56.457253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.457230 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:56.864683 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:10:56.864652 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58d1533_83ed_484d_9f32_5c4b54a10dea.slice/crio-ae500a83808895e50323674919c2ed199c3038e6e22e10b31c7ce9aedef956c4 WatchSource:0}: Error finding container ae500a83808895e50323674919c2ed199c3038e6e22e10b31c7ce9aedef956c4: Status 404 returned error can't find the container with id ae500a83808895e50323674919c2ed199c3038e6e22e10b31c7ce9aedef956c4 Oct 30 22:10:56.879496 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.879468 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd"] Oct 30 22:10:56.925920 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:10:56.925884 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6605229_d3aa_4a76_861c_4d429b9b4991.slice/crio-84a0feec7a3f0e4a6e3cda77b07631e08e88f4186e2f2b2f3477eac725365fcd WatchSource:0}: Error finding container 84a0feec7a3f0e4a6e3cda77b07631e08e88f4186e2f2b2f3477eac725365fcd: Status 404 returned error can't find the container with id 84a0feec7a3f0e4a6e3cda77b07631e08e88f4186e2f2b2f3477eac725365fcd Oct 30 22:10:56.927355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:56.927329 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8"] Oct 30 22:10:57.252646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.252612 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:10:57.252897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.252878 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:10:57.252947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.252927 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 22:10:57.431453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.422430 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" event={"ID":"d6605229-d3aa-4a76-861c-4d429b9b4991","Type":"ContainerStarted","Data":"7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8"} Oct 30 22:10:57.431453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.422502 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" event={"ID":"d6605229-d3aa-4a76-861c-4d429b9b4991","Type":"ContainerStarted","Data":"84a0feec7a3f0e4a6e3cda77b07631e08e88f4186e2f2b2f3477eac725365fcd"} Oct 30 22:10:57.431453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.423082 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:10:57.431453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.426039 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" event={"ID":"b58d1533-83ed-484d-9f32-5c4b54a10dea","Type":"ContainerStarted","Data":"21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07"} Oct 30 22:10:57.431453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.426082 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" event={"ID":"b58d1533-83ed-484d-9f32-5c4b54a10dea","Type":"ContainerStarted","Data":"ae500a83808895e50323674919c2ed199c3038e6e22e10b31c7ce9aedef956c4"} Oct 30 22:10:57.431453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.426110 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:10:57.565357 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.565253 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" podStartSLOduration=1.565234903 podStartE2EDuration="1.565234903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:57.558924028 +0000 UTC m=+1436.959480148" watchObservedRunningTime="2026-10-30 22:10:57.565234903 +0000 UTC m=+1436.965791023" Oct 30 22:10:57.685553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:57.685485 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" podStartSLOduration=1.685467316 podStartE2EDuration="1.685467316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:57.681925593 +0000 UTC m=+1437.082481727" watchObservedRunningTime="2026-10-30 22:10:57.685467316 +0000 UTC m=+1437.086023427" Oct 30 22:10:58.253516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:58.253482 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:10:58.253689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:58.253547 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 22:10:59.252859 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.252819 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:10:59.253303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.252896 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 22:10:59.491828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.491799 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b"] Oct 30 22:10:59.521302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.521228 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.556315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.556285 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b"] Oct 30 22:10:59.572253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.572224 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.572405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.572265 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.572497 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.572469 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.572629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.572604 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rffx\" (UniqueName: \"kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.622980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.622347 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-872730-serving-cert\"" Oct 30 22:10:59.673720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.673689 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.673903 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.673790 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rffx\" (UniqueName: \"kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.674063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.674002 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.674063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.674048 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.674236 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:10:59.674155 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-872730-serving-cert: secret "maas-api-e2e-api-a-872730-serving-cert" not found Oct 30 22:10:59.674285 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:10:59.674242 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls podName:06daf9c0-8241-440a-9770-e5040e3257d8 nodeName:}" failed. No retries permitted until 2026-10-30 22:11:00.174220331 +0000 UTC m=+1439.574776450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls") pod "maas-api-e2e-api-a-872730-84d975656b-fwm8b" (UID: "06daf9c0-8241-440a-9770-e5040e3257d8") : secret "maas-api-e2e-api-a-872730-serving-cert" not found Oct 30 22:10:59.674476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.674457 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.676621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.676602 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:10:59.750259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:10:59.750231 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rffx\" (UniqueName: \"kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:11:00.181078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:00.181042 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:11:00.183665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:00.183638 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\") pod \"maas-api-e2e-api-a-872730-84d975656b-fwm8b\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:11:00.253580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:00.253550 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:11:00.253983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:00.253599 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 22:11:00.437312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:00.437287 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:11:00.835501 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:11:00.835457 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06daf9c0_8241_440a_9770_e5040e3257d8.slice/crio-16c03c43f8a92575e5430e3da43bddaaa14c9f4ba9dd6cb676073d83777f6b57 WatchSource:0}: Error finding container 16c03c43f8a92575e5430e3da43bddaaa14c9f4ba9dd6cb676073d83777f6b57: Status 404 returned error can't find the container with id 16c03c43f8a92575e5430e3da43bddaaa14c9f4ba9dd6cb676073d83777f6b57 Oct 30 22:11:00.840099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:00.840074 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b"] Oct 30 22:11:01.253756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:01.253720 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:11:01.254281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:01.253785 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 22:11:01.457433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:01.457397 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" event={"ID":"06daf9c0-8241-440a-9770-e5040e3257d8","Type":"ContainerStarted","Data":"16c03c43f8a92575e5430e3da43bddaaa14c9f4ba9dd6cb676073d83777f6b57"} Oct 30 22:11:02.257176 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:02.257145 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:11:02.257593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:02.257571 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:11:02.258449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:02.258428 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:11:02.476665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:02.476623 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" event={"ID":"06daf9c0-8241-440a-9770-e5040e3257d8","Type":"ContainerStarted","Data":"b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af"} Oct 30 22:11:02.476838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:02.476719 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:11:02.686265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:02.686220 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" podStartSLOduration=2.723096387 podStartE2EDuration="3.686204806s" totalImagesPullingTime="963.108419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:02.681594053 +0000 UTC m=+1442.082150173" watchObservedRunningTime="2026-10-30 22:11:02.686204806 +0000 UTC m=+1442.086760926" Oct 30 22:11:08.437576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:08.437548 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:11:08.438039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:08.437616 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:11:08.489765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:08.489736 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:11:22.865094 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:11:22.865061 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:11:27.056096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:27.056061 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp"] Oct 30 22:11:27.106387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:27.106360 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4bb830-4xfvp"] Oct 30 22:11:28.871089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:28.871059 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd8cbe8-1932-41b2-b43b-a24eee44bf4b" path="/var/lib/kubelet/pods/8dd8cbe8-1932-41b2-b43b-a24eee44bf4b/volumes" Oct 30 22:11:32.051611 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:32.051581 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48"] Oct 30 22:11:32.102188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:32.102164 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e521d2-ngr48"] Oct 30 22:11:32.871850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:32.871819 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f" path="/var/lib/kubelet/pods/ae6a7c86-7b95-4f9a-8df6-4ccf7b99557f/volumes" Oct 30 22:11:33.771252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:33.771220 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp"] Oct 30 22:11:33.889214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:33.889185 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm"] Oct 30 22:11:38.772291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:38.772247 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" podUID="d453d922-38f1-49ca-a480-20543fa744fa" containerName="payload-processing" containerID="cri-o://642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4" gracePeriod=25 Oct 30 22:11:38.889579 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:38.889504 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" podUID="c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" containerName="payload-pre-processing" containerID="cri-o://168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4" gracePeriod=25 Oct 30 22:11:39.087423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.087392 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:11:39.163879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.163858 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:11:39.254384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.254355 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx\" (UniqueName: \"kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx\") pod \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " Oct 30 22:11:39.254539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.254524 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume\") pod \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\" (UID: \"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed\") " Oct 30 22:11:39.254670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.254654 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx\" (UniqueName: \"kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx\") pod \"d453d922-38f1-49ca-a480-20543fa744fa\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " Oct 30 22:11:39.254771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.254755 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume\") pod \"d453d922-38f1-49ca-a480-20543fa744fa\" (UID: \"d453d922-38f1-49ca-a480-20543fa744fa\") " Oct 30 22:11:39.254944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.254921 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume" pod "c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" (UID: "c0ba3daa-8dae-4413-b9f0-bdb34dd662ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:39.255366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.255334 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume" pod "d453d922-38f1-49ca-a480-20543fa744fa" (UID: "d453d922-38f1-49ca-a480-20543fa744fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:39.255366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.255355 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:39.256687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.256660 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx" pod "c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" (UID: "c0ba3daa-8dae-4413-b9f0-bdb34dd662ed"). InnerVolumeSpecName "kube-api-access-sxcrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:39.256892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.256866 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx" pod "d453d922-38f1-49ca-a480-20543fa744fa" (UID: "d453d922-38f1-49ca-a480-20543fa744fa"). InnerVolumeSpecName "kube-api-access-22tnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:39.356895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.356818 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tnx\" (UniqueName: \"kubernetes.io/projected/d453d922-38f1-49ca-a480-20543fa744fa-kube-api-access-22tnx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:39.356895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.356856 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d453d922-38f1-49ca-a480-20543fa744fa-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:39.356895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.356871 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxcrx\" (UniqueName: \"kubernetes.io/projected/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed-kube-api-access-sxcrx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:39.769081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.769046 5607 generic.go:356] "Generic (PLEG): container finished" podID="d453d922-38f1-49ca-a480-20543fa744fa" containerID="642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4" exitCode=0 Oct 30 22:11:39.769283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.769101 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" Oct 30 22:11:39.769283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.769125 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" event={"ID":"d453d922-38f1-49ca-a480-20543fa744fa","Type":"ContainerDied","Data":"642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4"} Oct 30 22:11:39.769283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.769217 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp" event={"ID":"d453d922-38f1-49ca-a480-20543fa744fa","Type":"ContainerDied","Data":"47097b96cefe24f6795655ffb81aabe036c4b6f4cdc4f2bc3c3cec5fb2e6812a"} Oct 30 22:11:39.769283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.769253 5607 scope.go:122] "RemoveContainer" containerID="642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4" Oct 30 22:11:39.770794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.770770 5607 generic.go:356] "Generic (PLEG): container finished" podID="c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" containerID="168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4" exitCode=0 Oct 30 22:11:39.770940 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.770824 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" Oct 30 22:11:39.770940 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.770900 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" event={"ID":"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed","Type":"ContainerDied","Data":"168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4"} Oct 30 22:11:39.771113 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.771014 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm" event={"ID":"c0ba3daa-8dae-4413-b9f0-bdb34dd662ed","Type":"ContainerDied","Data":"cdbffdbd12fc5220e368aa79eeecb51f748aa4a2491f691f01b7ef59870236f1"} Oct 30 22:11:39.789403 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.789061 5607 scope.go:122] "RemoveContainer" containerID="642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4" Oct 30 22:11:39.790113 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:11:39.790080 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4\": container with ID starting with 642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4 not found: ID does not exist" containerID="642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4" Oct 30 22:11:39.790240 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.790122 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4"} err="failed to get container status \"642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4\": rpc error: code = NotFound desc = could not find container \"642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4\": container with ID starting with 642b757fadd475e2b855b24ee6e48f2b12edc9c0009be9b4c25e3bcbe07d25f4 not found: ID does not exist" Oct 30 22:11:39.790240 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.790146 5607 scope.go:122] "RemoveContainer" containerID="168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4" Oct 30 22:11:39.809392 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.809373 5607 scope.go:122] "RemoveContainer" containerID="168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4" Oct 30 22:11:39.809690 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:11:39.809660 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4\": container with ID starting with 168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4 not found: ID does not exist" containerID="168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4" Oct 30 22:11:39.809787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.809699 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4"} err="failed to get container status \"168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4\": rpc error: code = NotFound desc = could not find container \"168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4\": container with ID starting with 168c38972b7e4b9553ca8b8a257590f830f7d920dd3539e09b90cf6c4e36a1e4 not found: ID does not exist" Oct 30 22:11:39.944749 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.944725 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm"] Oct 30 22:11:39.995729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:39.995705 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5e5df-6f9dccc475-rnnxm"] Oct 30 22:11:40.064431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:40.064345 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp"] Oct 30 22:11:40.068141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:40.068118 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5e5df-b849489b9-fcxbp"] Oct 30 22:11:40.872314 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:40.872285 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" path="/var/lib/kubelet/pods/c0ba3daa-8dae-4413-b9f0-bdb34dd662ed/volumes" Oct 30 22:11:40.872680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:40.872671 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d453d922-38f1-49ca-a480-20543fa744fa" path="/var/lib/kubelet/pods/d453d922-38f1-49ca-a480-20543fa744fa/volumes" Oct 30 22:11:43.057922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:43.057891 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79"] Oct 30 22:11:43.108530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:43.108502 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5824372f-nnt79"] Oct 30 22:11:44.871877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:44.871848 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393f38aa-e2f3-4771-9282-f3273194438c" path="/var/lib/kubelet/pods/393f38aa-e2f3-4771-9282-f3273194438c/volumes" Oct 30 22:11:52.864622 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:11:52.864579 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:11:58.725689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.725606 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx"] Oct 30 22:11:58.726631 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.726610 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d453d922-38f1-49ca-a480-20543fa744fa" containerName="payload-processing" Oct 30 22:11:58.726739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.726633 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d453d922-38f1-49ca-a480-20543fa744fa" containerName="payload-processing" Oct 30 22:11:58.726739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.726698 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" containerName="payload-pre-processing" Oct 30 22:11:58.726739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.726706 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" containerName="payload-pre-processing" Oct 30 22:11:58.726895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.726867 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d453d922-38f1-49ca-a480-20543fa744fa" containerName="payload-processing" Oct 30 22:11:58.726895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.726889 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ba3daa-8dae-4413-b9f0-bdb34dd662ed" containerName="payload-pre-processing" Oct 30 22:11:58.730810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.730789 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.788022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.787971 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4401bb-openshift-default-dockercfg-f6sb2\"" Oct 30 22:11:58.823582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.823544 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.823870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.823731 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.823870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.823788 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rv4g\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.824574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.823920 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.824574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.823998 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.824574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.824028 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.824574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.824075 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.824574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.824133 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.824574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.824267 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.856374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.856338 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx"] Oct 30 22:11:58.926137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926097 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rv4g\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.926368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926146 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.926368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926195 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.926368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926255 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.926368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926365 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.926781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926755 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.926946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.926923 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927264 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927322 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927433 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927506 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927646 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927799 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.927911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.927891 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.929463 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.929436 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.930933 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.930913 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.989630 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.989561 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rv4g\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:58.989769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:58.989719 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token\") pod \"e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:59.051395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.051360 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:11:59.477666 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:11:59.477637 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18c2dc3_2e1a_4647_9b5d_77c6d383f25b.slice/crio-6291f2ab3e1858fd50d3382435d711ed1344940abed08d0cdb5c7a19b6f146cb WatchSource:0}: Error finding container 6291f2ab3e1858fd50d3382435d711ed1344940abed08d0cdb5c7a19b6f146cb: Status 404 returned error can't find the container with id 6291f2ab3e1858fd50d3382435d711ed1344940abed08d0cdb5c7a19b6f146cb Oct 30 22:11:59.480401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.480365 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:59.480508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.480439 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:59.480508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.480466 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:59.482249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.482224 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx"] Oct 30 22:11:59.956214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.956183 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" event={"ID":"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b","Type":"ContainerStarted","Data":"90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b"} Oct 30 22:11:59.956651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:11:59.956243 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" event={"ID":"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b","Type":"ContainerStarted","Data":"6291f2ab3e1858fd50d3382435d711ed1344940abed08d0cdb5c7a19b6f146cb"} Oct 30 22:12:00.052331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.052305 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:12:00.058208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.058186 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:12:00.058711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.058699 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl"] Oct 30 22:12:00.062014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.061997 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a572ead-pbchl"] Oct 30 22:12:00.107154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.107113 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podStartSLOduration=2.107101912 podStartE2EDuration="2.107101912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:00.105318868 +0000 UTC m=+1499.505874990" watchObservedRunningTime="2026-10-30 22:12:00.107101912 +0000 UTC m=+1499.507658033" Oct 30 22:12:00.880844 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.880811 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4f1291-fc55-4d1c-a82e-758ca12a19d4" path="/var/lib/kubelet/pods/4d4f1291-fc55-4d1c-a82e-758ca12a19d4/volumes" Oct 30 22:12:00.962972 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.962926 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:12:00.963760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:00.963738 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:12:01.073453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:01.073429 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:12:01.077225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:01.077208 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:12:06.653092 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.653056 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8"] Oct 30 22:12:06.660135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.660111 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.721849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.721813 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8"] Oct 30 22:12:06.730757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.730725 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.731037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.731013 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brrd\" (UniqueName: \"kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd\") pod \"payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.767675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.767645 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4401bb-dockercfg-58l49\"" Oct 30 22:12:06.768013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.767994 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4401bb\"" Oct 30 22:12:06.829329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.829277 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9"] Oct 30 22:12:06.833004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.832672 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.833004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.832739 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brrd\" (UniqueName: \"kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd\") pod \"payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.833983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.833943 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.835597 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.835580 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:06.884163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.884133 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9"] Oct 30 22:12:06.896432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.896404 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brrd\" (UniqueName: \"kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd\") pod \"payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:06.934637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.934558 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4401bb-9675b947-89hg9\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:06.934796 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.934773 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shp2\" (UniqueName: \"kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2\") pod \"payload-processing-e2e-api-b-4401bb-9675b947-89hg9\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:06.977995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:06.975644 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:07.037193 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.037117 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4401bb-9675b947-89hg9\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:07.037348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.037249 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shp2\" (UniqueName: \"kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2\") pod \"payload-processing-e2e-api-b-4401bb-9675b947-89hg9\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:07.037943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.037914 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4401bb-9675b947-89hg9\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:07.100605 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.100578 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shp2\" (UniqueName: \"kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2\") pod \"payload-processing-e2e-api-b-4401bb-9675b947-89hg9\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:07.183470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.183442 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:07.360438 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:12:07.360404 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc718af35_daac_4738_9785_dc80e68982e3.slice/crio-23fb08ae7725c644f81b9b106c37b45739184b35ddd36cb54ecea90507951850 WatchSource:0}: Error finding container 23fb08ae7725c644f81b9b106c37b45739184b35ddd36cb54ecea90507951850: Status 404 returned error can't find the container with id 23fb08ae7725c644f81b9b106c37b45739184b35ddd36cb54ecea90507951850 Oct 30 22:12:07.363925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.363902 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8"] Oct 30 22:12:07.557729 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:12:07.557701 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258368e2_ecb9_4902_9924_87c599d7078b.slice/crio-9379ddcc6bd98a18dd276f5cf814a20186ca9d8fa0adb4ad32b43226439f9c8a WatchSource:0}: Error finding container 9379ddcc6bd98a18dd276f5cf814a20186ca9d8fa0adb4ad32b43226439f9c8a: Status 404 returned error can't find the container with id 9379ddcc6bd98a18dd276f5cf814a20186ca9d8fa0adb4ad32b43226439f9c8a Oct 30 22:12:07.561169 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:07.561148 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9"] Oct 30 22:12:08.019030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.018992 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" event={"ID":"c718af35-daac-4738-9785-dc80e68982e3","Type":"ContainerStarted","Data":"31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339"} Oct 30 22:12:08.019457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.019046 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" event={"ID":"c718af35-daac-4738-9785-dc80e68982e3","Type":"ContainerStarted","Data":"23fb08ae7725c644f81b9b106c37b45739184b35ddd36cb54ecea90507951850"} Oct 30 22:12:08.019457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.019077 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:08.020759 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.020735 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" event={"ID":"258368e2-ecb9-4902-9924-87c599d7078b","Type":"ContainerStarted","Data":"7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d"} Oct 30 22:12:08.020878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.020784 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:08.020878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.020809 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" event={"ID":"258368e2-ecb9-4902-9924-87c599d7078b","Type":"ContainerStarted","Data":"9379ddcc6bd98a18dd276f5cf814a20186ca9d8fa0adb4ad32b43226439f9c8a"} Oct 30 22:12:08.141494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.141452 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" podStartSLOduration=2.141438949 podStartE2EDuration="2.141438949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:08.1376392 +0000 UTC m=+1507.538195320" watchObservedRunningTime="2026-10-30 22:12:08.141438949 +0000 UTC m=+1507.541995070" Oct 30 22:12:08.260019 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:08.259976 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" podStartSLOduration=2.259939216 podStartE2EDuration="2.259939216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:08.256067964 +0000 UTC m=+1507.656624114" watchObservedRunningTime="2026-10-30 22:12:08.259939216 +0000 UTC m=+1507.660495364" Oct 30 22:12:19.031294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:19.031267 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:12:19.031700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:19.031381 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:12:22.864250 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:22.864216 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:12:24.053499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:24.053469 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz"] Oct 30 22:12:24.104925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:24.104895 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25be4af4-hksfz"] Oct 30 22:12:24.874230 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:24.874197 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19ce1e1-ff8f-4c92-bcce-404325e5f45a" path="/var/lib/kubelet/pods/f19ce1e1-ff8f-4c92-bcce-404325e5f45a/volumes" Oct 30 22:12:27.603772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:27.603742 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm"] Oct 30 22:12:27.724237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:27.724205 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6"] Oct 30 22:12:30.379527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.379491 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r"] Oct 30 22:12:30.398075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.379789 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" podUID="b9d258df-aeb2-48ab-98f1-1caf1a50b065" containerName="maas-api" containerID="cri-o://05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9" gracePeriod=30 Oct 30 22:12:30.739695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.738822 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:12:30.786724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.786695 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt\" (UniqueName: \"kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt\") pod \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " Oct 30 22:12:30.786898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.786872 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca\") pod \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " Oct 30 22:12:30.786975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.786936 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs\") pod \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " Oct 30 22:12:30.787146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.787124 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls\") pod \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\" (UID: \"b9d258df-aeb2-48ab-98f1-1caf1a50b065\") " Oct 30 22:12:30.787308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.787279 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca" pod "b9d258df-aeb2-48ab-98f1-1caf1a50b065" (UID: "b9d258df-aeb2-48ab-98f1-1caf1a50b065"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:30.788027 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.788001 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d258df-aeb2-48ab-98f1-1caf1a50b065-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:30.789558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.789532 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs" pod "b9d258df-aeb2-48ab-98f1-1caf1a50b065" (UID: "b9d258df-aeb2-48ab-98f1-1caf1a50b065"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:30.789558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.789544 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt" pod "b9d258df-aeb2-48ab-98f1-1caf1a50b065" (UID: "b9d258df-aeb2-48ab-98f1-1caf1a50b065"). InnerVolumeSpecName "kube-api-access-jcpnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:30.789710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.789576 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls" pod "b9d258df-aeb2-48ab-98f1-1caf1a50b065" (UID: "b9d258df-aeb2-48ab-98f1-1caf1a50b065"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:30.897002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.894836 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:30.897002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.894867 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcpnt\" (UniqueName: \"kubernetes.io/projected/b9d258df-aeb2-48ab-98f1-1caf1a50b065-kube-api-access-jcpnt\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:30.897002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:30.894883 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d258df-aeb2-48ab-98f1-1caf1a50b065-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:31.267632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.267592 5607 generic.go:356] "Generic (PLEG): container finished" podID="b9d258df-aeb2-48ab-98f1-1caf1a50b065" containerID="05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9" exitCode=0 Oct 30 22:12:31.267896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.267673 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" event={"ID":"b9d258df-aeb2-48ab-98f1-1caf1a50b065","Type":"ContainerDied","Data":"05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9"} Oct 30 22:12:31.267896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.267734 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" event={"ID":"b9d258df-aeb2-48ab-98f1-1caf1a50b065","Type":"ContainerDied","Data":"0f212ea5bea21dd02b051e0a7427bd62fee4e2695a1d59e0ed3d049dd0832657"} Oct 30 22:12:31.267896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.267760 5607 scope.go:122] "RemoveContainer" containerID="05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9" Oct 30 22:12:31.267896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.267896 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r" Oct 30 22:12:31.283911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.283890 5607 scope.go:122] "RemoveContainer" containerID="05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9" Oct 30 22:12:31.284402 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:31.284372 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9\": container with ID starting with 05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9 not found: ID does not exist" containerID="05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9" Oct 30 22:12:31.284567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.284412 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9"} err="failed to get container status \"05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9\": rpc error: code = NotFound desc = could not find container \"05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9\": container with ID starting with 05a1f004fdf03df601dce56ae55380086000d053ed469bd29b7986117b2a95a9 not found: ID does not exist" Oct 30 22:12:31.439912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.439872 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r"] Oct 30 22:12:31.490920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:31.490888 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5e5df-d555b68-57n7r"] Oct 30 22:12:32.604475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:32.604435 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" podUID="be841774-2081-4101-8f1d-e188cef9d660" containerName="payload-processing" containerID="cri-o://38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30" gracePeriod=25 Oct 30 22:12:32.724637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:32.724594 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerName="payload-pre-processing" containerID="cri-o://0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f" gracePeriod=25 Oct 30 22:12:32.872119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:32.872054 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d258df-aeb2-48ab-98f1-1caf1a50b065" path="/var/lib/kubelet/pods/b9d258df-aeb2-48ab-98f1-1caf1a50b065/volumes" Oct 30 22:12:33.392617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:33.392582 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" podUID="be841774-2081-4101-8f1d-e188cef9d660" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:33.392761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:33.392582 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:34.056148 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:34.056113 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs"] Oct 30 22:12:34.106615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:34.106589 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d80222-jqdgs"] Oct 30 22:12:34.872458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:34.872426 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d133f40-b85d-4929-8e2a-f1182a4bd9ac" path="/var/lib/kubelet/pods/3d133f40-b85d-4929-8e2a-f1182a4bd9ac/volumes" Oct 30 22:12:36.434172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.434130 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw"] Oct 30 22:12:36.434838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.434821 5607 cpu_manager.go:414] "containerMap: removing container" podUID="b9d258df-aeb2-48ab-98f1-1caf1a50b065" containerName="maas-api" Oct 30 22:12:36.434838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.434835 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d258df-aeb2-48ab-98f1-1caf1a50b065" containerName="maas-api" Oct 30 22:12:36.435083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.435013 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d258df-aeb2-48ab-98f1-1caf1a50b065" containerName="maas-api" Oct 30 22:12:36.440574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.440555 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.502377 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.502345 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-427d70-openshift-default-dockercfg-9mbht\"" Oct 30 22:12:36.552981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.552937 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw"] Oct 30 22:12:36.575119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575088 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.575284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575147 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.575284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575216 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.576283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575377 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.576283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575472 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.576283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575510 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.576283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575546 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.576283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575655 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.576283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.575736 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfx22\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677175 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677218 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677246 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677279 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677375 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677450 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfx22\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677554 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677578 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677600 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677640 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677675 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677692 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.677993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.677839 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.678294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.678276 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.680140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.680117 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.680288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.680246 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.746306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.746271 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.746576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.746541 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfx22\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22\") pod \"e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:36.755246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:36.755222 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:37.196203 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:12:37.196175 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af5df0b_401b_4bb9_bd6b_b6e91fe5d0e0.slice/crio-b47785c56d42a331ecf54d1d927b3cbb1f98448b9aabe48033dc89be4c2c31c3 WatchSource:0}: Error finding container b47785c56d42a331ecf54d1d927b3cbb1f98448b9aabe48033dc89be4c2c31c3: Status 404 returned error can't find the container with id b47785c56d42a331ecf54d1d927b3cbb1f98448b9aabe48033dc89be4c2c31c3 Oct 30 22:12:37.199128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.199096 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:37.199202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.199166 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:37.199202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.199195 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:37.199760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.199742 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw"] Oct 30 22:12:37.322734 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.322693 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" event={"ID":"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0","Type":"ContainerStarted","Data":"baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252"} Oct 30 22:12:37.322937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.322754 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" event={"ID":"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0","Type":"ContainerStarted","Data":"b47785c56d42a331ecf54d1d927b3cbb1f98448b9aabe48033dc89be4c2c31c3"} Oct 30 22:12:37.450413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.450308 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podStartSLOduration=1.450289889 podStartE2EDuration="1.450289889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:37.445704743 +0000 UTC m=+1536.846260864" watchObservedRunningTime="2026-10-30 22:12:37.450289889 +0000 UTC m=+1536.850846011" Oct 30 22:12:37.756288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:37.756207 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:38.761261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:38.761222 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:39.337007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:39.336976 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:39.338001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:39.337983 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:12:40.186596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:40.186501 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk"] Oct 30 22:12:40.187413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:40.187385 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" containerID="cri-o://1fbbda21afe90d34824b3f603359f80930dc8fc468b330d95333c93cba779d9e" gracePeriod=30 Oct 30 22:12:43.391719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:43.391680 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:43.392150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:43.391767 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" podUID="be841774-2081-4101-8f1d-e188cef9d660" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:44.247399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.247368 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk"] Oct 30 22:12:44.253505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.253485 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.311648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.311618 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk"] Oct 30 22:12:44.324149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.319248 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpbs\" (UniqueName: \"kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs\") pod \"payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.324149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.319348 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.357271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.357238 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-427d70\"" Oct 30 22:12:44.357439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.357333 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-427d70-dockercfg-7pr5v\"" Oct 30 22:12:44.423674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.423603 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpbs\" (UniqueName: \"kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs\") pod \"payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.424223 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.424137 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.425017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.424979 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf"] Oct 30 22:12:44.425461 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.425086 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.430255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.430237 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.462188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.462099 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf"] Oct 30 22:12:44.526880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.526788 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6zv\" (UniqueName: \"kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv\") pod \"payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.526880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.526838 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.534804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.534766 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpbs\" (UniqueName: \"kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs\") pod \"payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.569913 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.569872 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:44.628842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.628813 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6zv\" (UniqueName: \"kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv\") pod \"payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.629051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.628849 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.629814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.629789 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.696990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.691592 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6zv\" (UniqueName: \"kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv\") pod \"payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.747463 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.747432 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:44.958786 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:12:44.958754 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fc515f_55f6_49ac_815d_6b20ab486861.slice/crio-7bed2bd8e6c5f6b97a2dd0474fbae3407b52a12c4cb1084baf57a405d011e10e WatchSource:0}: Error finding container 7bed2bd8e6c5f6b97a2dd0474fbae3407b52a12c4cb1084baf57a405d011e10e: Status 404 returned error can't find the container with id 7bed2bd8e6c5f6b97a2dd0474fbae3407b52a12c4cb1084baf57a405d011e10e Oct 30 22:12:44.962708 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:44.962685 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk"] Oct 30 22:12:45.123572 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:12:45.123544 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60bc0c4_0d00_40c5_a610_0e247043568d.slice/crio-60ab3e7ef4d6dce84f15a2d5af1239a634321af866e1fc290a70c378a3c41a7a WatchSource:0}: Error finding container 60ab3e7ef4d6dce84f15a2d5af1239a634321af866e1fc290a70c378a3c41a7a: Status 404 returned error can't find the container with id 60ab3e7ef4d6dce84f15a2d5af1239a634321af866e1fc290a70c378a3c41a7a Oct 30 22:12:45.127114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.127090 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf"] Oct 30 22:12:45.453347 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.453212 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" event={"ID":"38fc515f-55f6-49ac-815d-6b20ab486861","Type":"ContainerStarted","Data":"adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853"} Oct 30 22:12:45.453347 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.453282 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" event={"ID":"38fc515f-55f6-49ac-815d-6b20ab486861","Type":"ContainerStarted","Data":"7bed2bd8e6c5f6b97a2dd0474fbae3407b52a12c4cb1084baf57a405d011e10e"} Oct 30 22:12:45.453347 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.453318 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:12:45.455728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.455702 5607 generic.go:356] "Generic (PLEG): container finished" podID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerID="1fbbda21afe90d34824b3f603359f80930dc8fc468b330d95333c93cba779d9e" exitCode=0 Oct 30 22:12:45.455834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.455779 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" event={"ID":"bb7d2999-08eb-4975-a1c9-d443d2313650","Type":"ContainerDied","Data":"1fbbda21afe90d34824b3f603359f80930dc8fc468b330d95333c93cba779d9e"} Oct 30 22:12:45.455911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.455834 5607 scope.go:122] "RemoveContainer" containerID="f64a9aecde9d48d4a138dd71dda4b8d2703cca558be64a7b61907cea8c4d1569" Oct 30 22:12:45.459094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.458756 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" event={"ID":"a60bc0c4-0d00-40c5-a610-0e247043568d","Type":"ContainerStarted","Data":"b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d"} Oct 30 22:12:45.459094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.458808 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" event={"ID":"a60bc0c4-0d00-40c5-a610-0e247043568d","Type":"ContainerStarted","Data":"60ab3e7ef4d6dce84f15a2d5af1239a634321af866e1fc290a70c378a3c41a7a"} Oct 30 22:12:45.459094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.458895 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:45.463745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.463725 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:12:45.549487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549461 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549584 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549675 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549696 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:45.549820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549713 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549761 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549824 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549866 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.549926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549905 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.550120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.549996 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data\") pod \"bb7d2999-08eb-4975-a1c9-d443d2313650\" (UID: \"bb7d2999-08eb-4975-a1c9-d443d2313650\") " Oct 30 22:12:45.550120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550041 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:45.550228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550126 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:45.550273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550262 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:45.550437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550405 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:45.551075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550738 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.551075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550761 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7d2999-08eb-4975-a1c9-d443d2313650-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.551075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550776 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.551075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550789 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.551075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.550804 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.552182 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.552159 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:45.552496 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.552476 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:45.552496 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.552486 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "kube-api-access-nwh94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:45.552624 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.552609 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo" pod "bb7d2999-08eb-4975-a1c9-d443d2313650" (UID: "bb7d2999-08eb-4975-a1c9-d443d2313650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:45.574209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.574165 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" podStartSLOduration=1.574153501 podStartE2EDuration="1.574153501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:45.571574695 +0000 UTC m=+1544.972130807" watchObservedRunningTime="2026-10-30 22:12:45.574153501 +0000 UTC m=+1544.974709622" Oct 30 22:12:45.651853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.651787 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwh94\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-kube-api-access-nwh94\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.651853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.651811 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.651853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.651826 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.651853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.651840 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7d2999-08eb-4975-a1c9-d443d2313650-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:45.690766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:45.690692 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" podStartSLOduration=1.690676251 podStartE2EDuration="1.690676251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:45.689300337 +0000 UTC m=+1545.089856455" watchObservedRunningTime="2026-10-30 22:12:45.690676251 +0000 UTC m=+1545.091232458" Oct 30 22:12:46.469239 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:46.469187 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" Oct 30 22:12:46.469765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:46.469222 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk" event={"ID":"bb7d2999-08eb-4975-a1c9-d443d2313650","Type":"ContainerDied","Data":"6e9b7bebd6ee55da4e7711a7ee2726712abe3418d63c674d8e643e54f29a8278"} Oct 30 22:12:46.469765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:46.469309 5607 scope.go:122] "RemoveContainer" containerID="1fbbda21afe90d34824b3f603359f80930dc8fc468b330d95333c93cba779d9e" Oct 30 22:12:46.710084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:46.710051 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk"] Oct 30 22:12:46.715228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:46.715206 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d5e5df-openshift-default-7c66456d47-b85lk"] Oct 30 22:12:46.871420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:46.871345 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" path="/var/lib/kubelet/pods/bb7d2999-08eb-4975-a1c9-d443d2313650/volumes" Oct 30 22:12:47.057620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:47.057591 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg"] Oct 30 22:12:47.061007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:47.060985 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d646474-9r2qg"] Oct 30 22:12:48.871449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:48.871421 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd10f42a-c43e-4b98-a0dc-5e9e2e507389" path="/var/lib/kubelet/pods/fd10f42a-c43e-4b98-a0dc-5e9e2e507389/volumes" Oct 30 22:12:52.642015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.641988 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr"] Oct 30 22:12:52.642718 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.642703 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" Oct 30 22:12:52.642760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.642721 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" Oct 30 22:12:52.642760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.642736 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" Oct 30 22:12:52.642760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.642743 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" Oct 30 22:12:52.642870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.642860 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" Oct 30 22:12:52.642913 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.642872 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d2999-08eb-4975-a1c9-d443d2313650" containerName="istio-proxy" Oct 30 22:12:52.646465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.646449 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.700309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.700278 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4401bb-serving-cert\"" Oct 30 22:12:52.736617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.736589 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.736729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.736632 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7swp\" (UniqueName: \"kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.736786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.736759 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.736841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.736826 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.758998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.756744 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr"] Oct 30 22:12:52.838989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.838935 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.839189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.839004 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7swp\" (UniqueName: \"kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.839189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.839046 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.839189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.839126 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.840211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.840173 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.840353 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:52.840294 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-serving-cert: secret "maas-api-e2e-api-b-4401bb-serving-cert" not found Oct 30 22:12:52.840436 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:52.840354 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls podName:a861c63d-9146-47ec-b1d3-9bb44de10feb nodeName:}" failed. No retries permitted until 2026-10-30 22:12:53.340333412 +0000 UTC m=+1552.740889524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls") pod "maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" (UID: "a861c63d-9146-47ec-b1d3-9bb44de10feb") : secret "maas-api-e2e-api-b-4401bb-serving-cert" not found Oct 30 22:12:52.859592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.859557 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.878337 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:52.878264 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:12:52.902603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.902565 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7swp\" (UniqueName: \"kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:52.983532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:52.983497 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:12:53.022339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.022310 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:12:53.049052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.049030 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume\") pod \"be841774-2081-4101-8f1d-e188cef9d660\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " Oct 30 22:12:53.049208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.049105 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8\" (UniqueName: \"kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8\") pod \"be841774-2081-4101-8f1d-e188cef9d660\" (UID: \"be841774-2081-4101-8f1d-e188cef9d660\") " Oct 30 22:12:53.049400 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.049380 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume" pod "be841774-2081-4101-8f1d-e188cef9d660" (UID: "be841774-2081-4101-8f1d-e188cef9d660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:53.049632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.049617 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be841774-2081-4101-8f1d-e188cef9d660-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:53.051270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.051250 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8" pod "be841774-2081-4101-8f1d-e188cef9d660" (UID: "be841774-2081-4101-8f1d-e188cef9d660"). InnerVolumeSpecName "kube-api-access-2z9t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:53.150894 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.150793 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume\") pod \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " Oct 30 22:12:53.151280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.151244 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v\" (UniqueName: \"kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v\") pod \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\" (UID: \"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3\") " Oct 30 22:12:53.151943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.151915 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9t8\" (UniqueName: \"kubernetes.io/projected/be841774-2081-4101-8f1d-e188cef9d660-kube-api-access-2z9t8\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:53.153998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.153943 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume" pod "bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" (UID: "bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:53.154697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.154664 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v" pod "bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" (UID: "bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3"). InnerVolumeSpecName "kube-api-access-hpp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:53.253723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.253689 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpp8v\" (UniqueName: \"kubernetes.io/projected/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-kube-api-access-hpp8v\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:53.253723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.253717 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:53.354677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.354642 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:53.357211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.357192 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\") pod \"maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:53.523323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.523292 5607 generic.go:356] "Generic (PLEG): container finished" podID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerID="0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f" exitCode=0 Oct 30 22:12:53.523491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.523340 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" Oct 30 22:12:53.523491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.523378 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" event={"ID":"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3","Type":"ContainerDied","Data":"0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f"} Oct 30 22:12:53.523491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.523442 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6" event={"ID":"bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3","Type":"ContainerDied","Data":"10bfb1787cb683221d38e582ee4ee41ca18796cab4e4f9b2966fe3b6f3c31b5f"} Oct 30 22:12:53.523491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.523472 5607 scope.go:122] "RemoveContainer" containerID="0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f" Oct 30 22:12:53.524890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.524867 5607 generic.go:356] "Generic (PLEG): container finished" podID="be841774-2081-4101-8f1d-e188cef9d660" containerID="38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30" exitCode=0 Oct 30 22:12:53.524997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.524914 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" Oct 30 22:12:53.524997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.524917 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" event={"ID":"be841774-2081-4101-8f1d-e188cef9d660","Type":"ContainerDied","Data":"38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30"} Oct 30 22:12:53.525110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.525057 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm" event={"ID":"be841774-2081-4101-8f1d-e188cef9d660","Type":"ContainerDied","Data":"aaa359ab8a2e300b718620e7655fef3ef3c1e44bd6b8c47e5f47ea78d6375692"} Oct 30 22:12:53.535354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.535340 5607 scope.go:122] "RemoveContainer" containerID="0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f" Oct 30 22:12:53.535644 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:53.535621 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f\": container with ID starting with 0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f not found: ID does not exist" containerID="0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f" Oct 30 22:12:53.535702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.535654 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f"} err="failed to get container status \"0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f\": rpc error: code = NotFound desc = could not find container \"0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f\": container with ID starting with 0b56d746c28a06e9821d5ac284d39ad81d0c63522f6e8ed9a637ca515d194a9f not found: ID does not exist" Oct 30 22:12:53.535702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.535681 5607 scope.go:122] "RemoveContainer" containerID="38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30" Oct 30 22:12:53.545278 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.545263 5607 scope.go:122] "RemoveContainer" containerID="38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30" Oct 30 22:12:53.545550 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:12:53.545531 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30\": container with ID starting with 38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30 not found: ID does not exist" containerID="38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30" Oct 30 22:12:53.545593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.545557 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30"} err="failed to get container status \"38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30\": rpc error: code = NotFound desc = could not find container \"38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30\": container with ID starting with 38afa55092be04a0108c46421027f466ea4d51fba41a01771be6658f89365e30 not found: ID does not exist" Oct 30 22:12:53.627746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.627726 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:53.705001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.700156 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm"] Oct 30 22:12:53.705001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.704199 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-d8wlm"] Oct 30 22:12:53.814061 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.813987 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6"] Oct 30 22:12:53.817301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:53.817279 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-87rp6"] Oct 30 22:12:54.016181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.016155 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr"] Oct 30 22:12:54.017266 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:12:54.017240 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda861c63d_9146_47ec_b1d3_9bb44de10feb.slice/crio-b191a6ed582b4324ee0b77b475ad05a251fd66eacf276d46504dd3dda4c57d21 WatchSource:0}: Error finding container b191a6ed582b4324ee0b77b475ad05a251fd66eacf276d46504dd3dda4c57d21: Status 404 returned error can't find the container with id b191a6ed582b4324ee0b77b475ad05a251fd66eacf276d46504dd3dda4c57d21 Oct 30 22:12:54.059021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.058991 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p"] Oct 30 22:12:54.062678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.062648 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd92762-dxv2p"] Oct 30 22:12:54.535774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.535737 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" event={"ID":"a861c63d-9146-47ec-b1d3-9bb44de10feb","Type":"ContainerStarted","Data":"b191a6ed582b4324ee0b77b475ad05a251fd66eacf276d46504dd3dda4c57d21"} Oct 30 22:12:54.874122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.874052 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5fecfd-4e66-4517-95bb-beff04eb037e" path="/var/lib/kubelet/pods/2e5fecfd-4e66-4517-95bb-beff04eb037e/volumes" Oct 30 22:12:54.874474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.874453 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" path="/var/lib/kubelet/pods/bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3/volumes" Oct 30 22:12:54.874809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:54.874795 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be841774-2081-4101-8f1d-e188cef9d660" path="/var/lib/kubelet/pods/be841774-2081-4101-8f1d-e188cef9d660/volumes" Oct 30 22:12:55.219979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:55.219923 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2"] Oct 30 22:12:55.345204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:55.345171 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl"] Oct 30 22:12:55.547088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:55.546992 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" event={"ID":"a861c63d-9146-47ec-b1d3-9bb44de10feb","Type":"ContainerStarted","Data":"dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6"} Oct 30 22:12:55.547326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:55.547305 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:12:55.669200 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:55.669137 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" podStartSLOduration=2.75365319 podStartE2EDuration="3.669115613s" totalImagesPullingTime="915.462423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:55.666736611 +0000 UTC m=+1555.067292735" watchObservedRunningTime="2026-10-30 22:12:55.669115613 +0000 UTC m=+1555.069671736" Oct 30 22:12:56.471970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:56.471927 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:12:56.472792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:12:56.472772 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:13:00.220356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.220317 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" podUID="8b74249f-5fd2-4d43-aabe-89200c159d60" containerName="payload-processing" containerID="cri-o://5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0" gracePeriod=25 Oct 30 22:13:00.346323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.346291 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" podUID="a65fa155-efe9-4f4e-b149-101ec99272f9" containerName="payload-pre-processing" containerID="cri-o://262df72b2da409843bdf8ff98beec0a348c215bb629b52b5c3bdd896910565d6" gracePeriod=25 Oct 30 22:13:00.522715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.522684 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:13:00.592534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.592501 5607 generic.go:356] "Generic (PLEG): container finished" podID="a65fa155-efe9-4f4e-b149-101ec99272f9" containerID="262df72b2da409843bdf8ff98beec0a348c215bb629b52b5c3bdd896910565d6" exitCode=0 Oct 30 22:13:00.592700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.592572 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" event={"ID":"a65fa155-efe9-4f4e-b149-101ec99272f9","Type":"ContainerDied","Data":"262df72b2da409843bdf8ff98beec0a348c215bb629b52b5c3bdd896910565d6"} Oct 30 22:13:00.594280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.594256 5607 generic.go:356] "Generic (PLEG): container finished" podID="8b74249f-5fd2-4d43-aabe-89200c159d60" containerID="5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0" exitCode=0 Oct 30 22:13:00.594474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.594318 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" Oct 30 22:13:00.594561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.594334 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" event={"ID":"8b74249f-5fd2-4d43-aabe-89200c159d60","Type":"ContainerDied","Data":"5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0"} Oct 30 22:13:00.594630 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.594600 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2" event={"ID":"8b74249f-5fd2-4d43-aabe-89200c159d60","Type":"ContainerDied","Data":"8bc23d9db40fdad8175903ee28973fc537054374e504373eaba005f055655967"} Oct 30 22:13:00.594675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.594635 5607 scope.go:122] "RemoveContainer" containerID="5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0" Oct 30 22:13:00.606286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.606268 5607 scope.go:122] "RemoveContainer" containerID="5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0" Oct 30 22:13:00.606607 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:13:00.606577 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0\": container with ID starting with 5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0 not found: ID does not exist" containerID="5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0" Oct 30 22:13:00.606706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.606617 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0"} err="failed to get container status \"5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0\": rpc error: code = NotFound desc = could not find container \"5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0\": container with ID starting with 5ed8f88a2afdd90aa8f99b94bf33b74568f29b63e47d8c1c0e1c8cbfadecafb0 not found: ID does not exist" Oct 30 22:13:00.660485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.660459 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2\" (UniqueName: \"kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2\") pod \"8b74249f-5fd2-4d43-aabe-89200c159d60\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " Oct 30 22:13:00.660767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.660750 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume\") pod \"8b74249f-5fd2-4d43-aabe-89200c159d60\" (UID: \"8b74249f-5fd2-4d43-aabe-89200c159d60\") " Oct 30 22:13:00.661547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.661524 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume" pod "8b74249f-5fd2-4d43-aabe-89200c159d60" (UID: "8b74249f-5fd2-4d43-aabe-89200c159d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:00.663925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.663904 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:13:00.664104 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.664072 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2" pod "8b74249f-5fd2-4d43-aabe-89200c159d60" (UID: "8b74249f-5fd2-4d43-aabe-89200c159d60"). InnerVolumeSpecName "kube-api-access-c2rq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:00.761999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.761920 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume\") pod \"a65fa155-efe9-4f4e-b149-101ec99272f9\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " Oct 30 22:13:00.762123 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.762052 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh\" (UniqueName: \"kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh\") pod \"a65fa155-efe9-4f4e-b149-101ec99272f9\" (UID: \"a65fa155-efe9-4f4e-b149-101ec99272f9\") " Oct 30 22:13:00.762284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.762262 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume" pod "a65fa155-efe9-4f4e-b149-101ec99272f9" (UID: "a65fa155-efe9-4f4e-b149-101ec99272f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:00.762536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.762521 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b74249f-5fd2-4d43-aabe-89200c159d60-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:00.762588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.762539 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rq2\" (UniqueName: \"kubernetes.io/projected/8b74249f-5fd2-4d43-aabe-89200c159d60-kube-api-access-c2rq2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:00.762588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.762549 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65fa155-efe9-4f4e-b149-101ec99272f9-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:00.764256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.764234 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh" pod "a65fa155-efe9-4f4e-b149-101ec99272f9" (UID: "a65fa155-efe9-4f4e-b149-101ec99272f9"). InnerVolumeSpecName "kube-api-access-bg6vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:00.863967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:00.863916 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6vh\" (UniqueName: \"kubernetes.io/projected/a65fa155-efe9-4f4e-b149-101ec99272f9-kube-api-access-bg6vh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:13:01.018054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.017979 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2"] Oct 30 22:13:01.023251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.023227 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbe0a7-6798b9b4f5-5cjf2"] Oct 30 22:13:01.052711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.052687 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr"] Oct 30 22:13:01.064499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.064477 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48dd9313-7h4mr"] Oct 30 22:13:01.358185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.358133 5607 scope.go:122] "RemoveContainer" containerID="12cb02e039f9b6cbab66d0a81749500b98da06eed689e0b4db1e85ab0ea2dffd" Oct 30 22:13:01.368019 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.368001 5607 scope.go:122] "RemoveContainer" containerID="79992eacd434db60536d10b0cc5dc0c6d00e8c097a38315118f2e7dc1c2d0afa" Oct 30 22:13:01.377156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.377134 5607 scope.go:122] "RemoveContainer" containerID="135fcddfbc9ecaa17a0fb5e898806ddda177657c797aa5cc84dafc32ae2d82fd" Oct 30 22:13:01.385824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.385808 5607 scope.go:122] "RemoveContainer" containerID="6e38b6df3bb0962bb9b1efd65d8d36bab590a6d289c91e8def2ca0f4b3a93b38" Oct 30 22:13:01.394401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.394386 5607 scope.go:122] "RemoveContainer" containerID="2508c8aaf9e02f88ba9990f0742f29f7e8163be172dc6390fe03fcaa5a9445c8" Oct 30 22:13:01.560556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.560533 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:13:01.603473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.603444 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" Oct 30 22:13:01.603473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.603454 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl" event={"ID":"a65fa155-efe9-4f4e-b149-101ec99272f9","Type":"ContainerDied","Data":"f62c64e4a28966fcc345159f211157870ac1117888fe9d73888feb38a93202e9"} Oct 30 22:13:01.603652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.603534 5607 scope.go:122] "RemoveContainer" containerID="262df72b2da409843bdf8ff98beec0a348c215bb629b52b5c3bdd896910565d6" Oct 30 22:13:01.807503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.807468 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl"] Oct 30 22:13:01.810887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:01.810863 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dbe0a7-5ccb8dbc47-gwcdl"] Oct 30 22:13:02.871947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:02.871914 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b74249f-5fd2-4d43-aabe-89200c159d60" path="/var/lib/kubelet/pods/8b74249f-5fd2-4d43-aabe-89200c159d60/volumes" Oct 30 22:13:02.872443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:02.872427 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a" path="/var/lib/kubelet/pods/8e627bb5-1d9e-47ed-bc1a-90f5fd5a682a/volumes" Oct 30 22:13:02.872841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:02.872827 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65fa155-efe9-4f4e-b149-101ec99272f9" path="/var/lib/kubelet/pods/a65fa155-efe9-4f4e-b149-101ec99272f9/volumes" Oct 30 22:13:06.051155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:06.051126 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7"] Oct 30 22:13:06.102046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:06.102020 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ec7da7-lplb7"] Oct 30 22:13:06.870877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:06.870837 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7a9432-99ab-46c9-a2eb-061b13d0548d" path="/var/lib/kubelet/pods/3f7a9432-99ab-46c9-a2eb-061b13d0548d/volumes" Oct 30 22:13:07.052318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:07.052289 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg"] Oct 30 22:13:07.102915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:07.102888 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9ef690-zqkxg"] Oct 30 22:13:08.784012 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.783936 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds"] Oct 30 22:13:08.784689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784675 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8b74249f-5fd2-4d43-aabe-89200c159d60" containerName="payload-processing" Oct 30 22:13:08.784738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784691 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b74249f-5fd2-4d43-aabe-89200c159d60" containerName="payload-processing" Oct 30 22:13:08.784738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784705 5607 cpu_manager.go:414] "containerMap: removing container" podUID="be841774-2081-4101-8f1d-e188cef9d660" containerName="payload-processing" Oct 30 22:13:08.784738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784710 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be841774-2081-4101-8f1d-e188cef9d660" containerName="payload-processing" Oct 30 22:13:08.784738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784723 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerName="payload-pre-processing" Oct 30 22:13:08.784738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784728 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerName="payload-pre-processing" Oct 30 22:13:08.784889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784750 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a65fa155-efe9-4f4e-b149-101ec99272f9" containerName="payload-pre-processing" Oct 30 22:13:08.784889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784755 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65fa155-efe9-4f4e-b149-101ec99272f9" containerName="payload-pre-processing" Oct 30 22:13:08.784889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784858 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="be841774-2081-4101-8f1d-e188cef9d660" containerName="payload-processing" Oct 30 22:13:08.784889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784870 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7c339f-ddf5-4cfc-82b0-8dd44dc262a3" containerName="payload-pre-processing" Oct 30 22:13:08.784889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784879 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65fa155-efe9-4f4e-b149-101ec99272f9" containerName="payload-pre-processing" Oct 30 22:13:08.784889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.784889 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b74249f-5fd2-4d43-aabe-89200c159d60" containerName="payload-processing" Oct 30 22:13:08.790673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.790652 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.853800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.845710 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-04d404-openshift-default-dockercfg-dl2hb\"" Oct 30 22:13:08.871329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871285 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871357 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871464 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871505 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgc8\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871563 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871702 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871764 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871852 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.871934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.871890 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.892597 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.892561 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d466542-974b-4bb6-9d66-5985588eca3e" path="/var/lib/kubelet/pods/8d466542-974b-4bb6-9d66-5985588eca3e/volumes" Oct 30 22:13:08.909623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.909572 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds"] Oct 30 22:13:08.973787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.973748 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.973811 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.973880 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.973915 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974035 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974073 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgc8\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974111 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974238 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974303 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974479 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974565 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974695 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.974920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.974899 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.975183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.975164 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.977409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.977386 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:08.978487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:08.978467 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:09.036973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.036879 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:09.036973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.036933 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgc8\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8\") pod \"e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:09.107693 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.107662 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:09.544827 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:13:09.544785 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551af49d_9c92_48ed_9a8f_831c634c31bd.slice/crio-6e60822e987a90a48352d54217b52bd499ae3db3f318770d0509e9b187491fb0 WatchSource:0}: Error finding container 6e60822e987a90a48352d54217b52bd499ae3db3f318770d0509e9b187491fb0: Status 404 returned error can't find the container with id 6e60822e987a90a48352d54217b52bd499ae3db3f318770d0509e9b187491fb0 Oct 30 22:13:09.547784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.547739 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:09.547885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.547852 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:09.547930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.547895 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:09.549372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.549349 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds"] Oct 30 22:13:09.669484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.669444 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" event={"ID":"551af49d-9c92-48ed-9a8f-831c634c31bd","Type":"ContainerStarted","Data":"f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c"} Oct 30 22:13:09.669639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:09.669507 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" event={"ID":"551af49d-9c92-48ed-9a8f-831c634c31bd","Type":"ContainerStarted","Data":"6e60822e987a90a48352d54217b52bd499ae3db3f318770d0509e9b187491fb0"} Oct 30 22:13:10.108404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:10.108365 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:10.110027 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:10.109999 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:13:10.110193 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:10.110050 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 22:13:11.124382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:11.124344 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:11.247485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:11.247426 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podStartSLOduration=3.247412981 podStartE2EDuration="3.247412981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:09.796328294 +0000 UTC m=+1569.196884415" watchObservedRunningTime="2026-10-30 22:13:11.247412981 +0000 UTC m=+1570.647969102" Oct 30 22:13:11.684458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:11.684426 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:11.685404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:11.685388 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:13:18.619981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.619323 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf"] Oct 30 22:13:18.626679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.626029 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.684005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.683465 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04d404\"" Oct 30 22:13:18.735620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.735581 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf"] Oct 30 22:13:18.735855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.735754 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04d404-dockercfg-4zqrt\"" Oct 30 22:13:18.737739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.737704 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsl9l\" (UniqueName: \"kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l\") pod \"payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.752125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.747231 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.850613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.849113 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsl9l\" (UniqueName: \"kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l\") pod \"payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.850613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.849275 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.850613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.850293 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.913358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.913292 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsl9l\" (UniqueName: \"kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l\") pod \"payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:18.947463 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:18.947433 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:19.322516 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:13:19.322483 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232b4e73_0623_42a6_a244_90443cb59552.slice/crio-3a6b9347c940aa6dd62d77955de5cc35af57a4615d54c6e79453ad864d17b15b WatchSource:0}: Error finding container 3a6b9347c940aa6dd62d77955de5cc35af57a4615d54c6e79453ad864d17b15b: Status 404 returned error can't find the container with id 3a6b9347c940aa6dd62d77955de5cc35af57a4615d54c6e79453ad864d17b15b Oct 30 22:13:19.325837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:19.325744 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf"] Oct 30 22:13:19.802755 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:19.802713 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" event={"ID":"232b4e73-0623-42a6-a244-90443cb59552","Type":"ContainerStarted","Data":"0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86"} Oct 30 22:13:19.803348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:19.802790 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" event={"ID":"232b4e73-0623-42a6-a244-90443cb59552","Type":"ContainerStarted","Data":"3a6b9347c940aa6dd62d77955de5cc35af57a4615d54c6e79453ad864d17b15b"} Oct 30 22:13:19.803348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:19.803060 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:19.931566 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:19.931508 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" podStartSLOduration=1.931490599 podStartE2EDuration="1.931490599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:19.929732729 +0000 UTC m=+1579.330288851" watchObservedRunningTime="2026-10-30 22:13:19.931490599 +0000 UTC m=+1579.332046724" Oct 30 22:13:20.812150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:20.812115 5607 generic.go:356] "Generic (PLEG): container finished" podID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerID="90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b" exitCode=0 Oct 30 22:13:20.812649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:20.812183 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" event={"ID":"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b","Type":"ContainerDied","Data":"90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b"} Oct 30 22:13:20.812649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:20.812546 5607 scope.go:122] "RemoveContainer" containerID="90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b" Oct 30 22:13:20.814015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:20.813984 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:20.814133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:20.814072 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:20.814133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:20.814123 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:21.830888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:21.830853 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" event={"ID":"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b","Type":"ContainerStarted","Data":"57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f"} Oct 30 22:13:22.052107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:22.052073 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:13:22.052362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:22.052296 5607 patch_prober.go:29] interesting pod/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:13:22.052362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:22.052350 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:13:22.865462 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:13:22.865429 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:13:23.052156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:23.052122 5607 patch_prober.go:29] interesting pod/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:13:23.052330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:23.052179 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:13:24.051682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:24.051648 5607 patch_prober.go:29] interesting pod/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:13:24.052117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:24.051727 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:13:24.052117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:24.051993 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw"] Oct 30 22:13:24.122478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:24.122441 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f424bc8-9d2lw"] Oct 30 22:13:24.871800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:24.871768 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e612835-bbfc-4ee4-aa6d-a5081df60854" path="/var/lib/kubelet/pods/5e612835-bbfc-4ee4-aa6d-a5081df60854/volumes" Oct 30 22:13:25.052510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:25.052482 5607 patch_prober.go:29] interesting pod/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:13:25.052903 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:25.052537 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:13:26.052312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:26.052276 5607 patch_prober.go:29] interesting pod/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:13:26.052482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:26.052332 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 22:13:27.056763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:27.056733 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:13:27.057265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:27.057243 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:13:27.058083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:27.058062 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:13:30.816219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:30.816190 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:13:35.053153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:35.053123 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt"] Oct 30 22:13:35.104333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:35.104307 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c14b5fc-kk5nt"] Oct 30 22:13:36.879742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:36.879696 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21596c24-589d-4d5e-9448-c219940f0342" path="/var/lib/kubelet/pods/21596c24-589d-4d5e-9448-c219940f0342/volumes" Oct 30 22:13:52.864020 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:13:52.863939 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:13:55.372773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:55.372741 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5"] Oct 30 22:13:55.373255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:13:55.373021 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" containerID="cri-o://3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a" gracePeriod=30 Oct 30 22:14:00.650371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.650316 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:14:00.730455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730431 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730595 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730481 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730595 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730505 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730607 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730643 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730702 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730753 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.730864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730773 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.730864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730827 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.731067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730885 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo\") pod \"00710edf-10cd-4588-8fc4-68fdb78c78e9\" (UID: \"00710edf-10cd-4588-8fc4-68fdb78c78e9\") " Oct 30 22:14:00.731067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730926 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:00.731067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730984 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.731067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.730994 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.731284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.731085 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.731535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.731510 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00710edf-10cd-4588-8fc4-68fdb78c78e9-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.731802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.731540 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.731802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.731554 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.731802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.731566 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.731802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.731580 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.733237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.733201 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "kube-api-access-f7jzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.733237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.733228 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:00.733453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.733432 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:00.733525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.733508 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token" pod "00710edf-10cd-4588-8fc4-68fdb78c78e9" (UID: "00710edf-10cd-4588-8fc4-68fdb78c78e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:00.845626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.844264 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.845626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.844296 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.845626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.844312 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jzd\" (UniqueName: \"kubernetes.io/projected/00710edf-10cd-4588-8fc4-68fdb78c78e9-kube-api-access-f7jzd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:00.845626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:00.844326 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00710edf-10cd-4588-8fc4-68fdb78c78e9-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.147511 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.147472 5607 generic.go:356] "Generic (PLEG): container finished" podID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerID="3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a" exitCode=0 Oct 30 22:14:01.147680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.147573 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" Oct 30 22:14:01.147680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.147609 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" event={"ID":"00710edf-10cd-4588-8fc4-68fdb78c78e9","Type":"ContainerDied","Data":"3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a"} Oct 30 22:14:01.147680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.147662 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5" event={"ID":"00710edf-10cd-4588-8fc4-68fdb78c78e9","Type":"ContainerDied","Data":"f6d494ae92e3ffe60bbdb93015e80d7513a35131ac0a92504b65bb869b5421f1"} Oct 30 22:14:01.147809 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.147684 5607 scope.go:122] "RemoveContainer" containerID="3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a" Oct 30 22:14:01.161393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.161357 5607 scope.go:122] "RemoveContainer" containerID="3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a" Oct 30 22:14:01.161903 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:01.161878 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a\": container with ID starting with 3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a not found: ID does not exist" containerID="3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a" Oct 30 22:14:01.162078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.161909 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a"} err="failed to get container status \"3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a\": rpc error: code = NotFound desc = could not find container \"3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a\": container with ID starting with 3c6645ea0032731b72dd2da5101d9db263c47e630c35cddc5209793fcf34a22a not found: ID does not exist" Oct 30 22:14:01.318471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.318436 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5"] Oct 30 22:14:01.369667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.369633 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-xtwf5"] Oct 30 22:14:01.465856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.465830 5607 scope.go:122] "RemoveContainer" containerID="a6df4fe67246553d025f623d7516ce66692350babef6d37dd787ed28adbf45d7" Oct 30 22:14:01.476404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.476383 5607 scope.go:122] "RemoveContainer" containerID="652ee63540357d6adf9a182422187a3c34c7adfa43bd320aee5c51d76bca09fd" Oct 30 22:14:01.485388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.485370 5607 scope.go:122] "RemoveContainer" containerID="5b0fb8370793d74da4f2f44987e2d91c409790eee317b2b6195eeb6286f3f5a5" Oct 30 22:14:01.494084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.493897 5607 scope.go:122] "RemoveContainer" containerID="e713dbc7e4422006ecd42a0ff4250a5f6a82486af09188074e02386a7051ff8b" Oct 30 22:14:01.502466 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.502449 5607 scope.go:122] "RemoveContainer" containerID="db1a5b83f958e08f51c9b12dfed1fe6b22e2bd0b121d4bb5850d3e5b473f02b5" Oct 30 22:14:01.511009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:01.510991 5607 scope.go:122] "RemoveContainer" containerID="75d3affd72c423e0aab304d91af53d2fb57836f289c4de4ea05ec80bbf44d7ee" Oct 30 22:14:02.871787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:02.871753 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" path="/var/lib/kubelet/pods/00710edf-10cd-4588-8fc4-68fdb78c78e9/volumes" Oct 30 22:14:04.011192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:04.011158 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr"] Oct 30 22:14:04.117468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:04.117436 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv"] Oct 30 22:14:08.949735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:08.949691 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md"] Oct 30 22:14:08.950364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:08.949988 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" podUID="a391201c-a4d6-454b-b829-060b7dc6d5c9" containerName="maas-api" containerID="cri-o://94a1a021fdbbddb7e044adff367f4e7fafac387bfeaa6d98af391d5a3e340812" gracePeriod=30 Oct 30 22:14:09.027112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.027053 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" podUID="252cf1fd-2366-4e63-bcd0-e1af768fec53" containerName="payload-processing" containerID="cri-o://f82de597e0f7abb502673b969def000f9ba7b5d50849e6c8a8f76b723f1e118b" gracePeriod=25 Oct 30 22:14:09.124013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.123412 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" podUID="66ba75ef-f1f3-44c0-b296-8668a1c514af" containerName="payload-pre-processing" containerID="cri-o://d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981" gracePeriod=25 Oct 30 22:14:09.266364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.266315 5607 generic.go:356] "Generic (PLEG): container finished" podID="a391201c-a4d6-454b-b829-060b7dc6d5c9" containerID="94a1a021fdbbddb7e044adff367f4e7fafac387bfeaa6d98af391d5a3e340812" exitCode=0 Oct 30 22:14:09.266673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.266522 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" event={"ID":"a391201c-a4d6-454b-b829-060b7dc6d5c9","Type":"ContainerDied","Data":"94a1a021fdbbddb7e044adff367f4e7fafac387bfeaa6d98af391d5a3e340812"} Oct 30 22:14:09.268436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.268359 5607 generic.go:356] "Generic (PLEG): container finished" podID="252cf1fd-2366-4e63-bcd0-e1af768fec53" containerID="f82de597e0f7abb502673b969def000f9ba7b5d50849e6c8a8f76b723f1e118b" exitCode=0 Oct 30 22:14:09.268557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.268422 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" event={"ID":"252cf1fd-2366-4e63-bcd0-e1af768fec53","Type":"ContainerDied","Data":"f82de597e0f7abb502673b969def000f9ba7b5d50849e6c8a8f76b723f1e118b"} Oct 30 22:14:09.342981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.342842 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:14:09.438015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.437462 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp\" (UniqueName: \"kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp\") pod \"a391201c-a4d6-454b-b829-060b7dc6d5c9\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " Oct 30 22:14:09.438015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.437540 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca\") pod \"a391201c-a4d6-454b-b829-060b7dc6d5c9\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " Oct 30 22:14:09.438015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.437685 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs\") pod \"a391201c-a4d6-454b-b829-060b7dc6d5c9\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " Oct 30 22:14:09.438015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.437723 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\") pod \"a391201c-a4d6-454b-b829-060b7dc6d5c9\" (UID: \"a391201c-a4d6-454b-b829-060b7dc6d5c9\") " Oct 30 22:14:09.454987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.447367 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca" pod "a391201c-a4d6-454b-b829-060b7dc6d5c9" (UID: "a391201c-a4d6-454b-b829-060b7dc6d5c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:09.465765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.465668 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls" pod "a391201c-a4d6-454b-b829-060b7dc6d5c9" (UID: "a391201c-a4d6-454b-b829-060b7dc6d5c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:09.500234 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.489143 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp" pod "a391201c-a4d6-454b-b829-060b7dc6d5c9" (UID: "a391201c-a4d6-454b-b829-060b7dc6d5c9"). InnerVolumeSpecName "kube-api-access-kkksp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:09.551980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.551832 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs" pod "a391201c-a4d6-454b-b829-060b7dc6d5c9" (UID: "a391201c-a4d6-454b-b829-060b7dc6d5c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:09.553814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.553641 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkksp\" (UniqueName: \"kubernetes.io/projected/a391201c-a4d6-454b-b829-060b7dc6d5c9-kube-api-access-kkksp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.553814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.553673 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a391201c-a4d6-454b-b829-060b7dc6d5c9-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.553814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.553691 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.553814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.553705 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a391201c-a4d6-454b-b829-060b7dc6d5c9-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.565049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.565022 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:14:09.654921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.654891 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46\" (UniqueName: \"kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46\") pod \"252cf1fd-2366-4e63-bcd0-e1af768fec53\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " Oct 30 22:14:09.655126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.655057 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume\") pod \"252cf1fd-2366-4e63-bcd0-e1af768fec53\" (UID: \"252cf1fd-2366-4e63-bcd0-e1af768fec53\") " Oct 30 22:14:09.655667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.655636 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume" pod "252cf1fd-2366-4e63-bcd0-e1af768fec53" (UID: "252cf1fd-2366-4e63-bcd0-e1af768fec53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:09.658345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.658320 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46" pod "252cf1fd-2366-4e63-bcd0-e1af768fec53" (UID: "252cf1fd-2366-4e63-bcd0-e1af768fec53"). InnerVolumeSpecName "kube-api-access-2tv46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:09.686678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.686659 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:14:09.756663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.756586 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk\" (UniqueName: \"kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk\") pod \"66ba75ef-f1f3-44c0-b296-8668a1c514af\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " Oct 30 22:14:09.756815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.756749 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume\") pod \"66ba75ef-f1f3-44c0-b296-8668a1c514af\" (UID: \"66ba75ef-f1f3-44c0-b296-8668a1c514af\") " Oct 30 22:14:09.757152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.757125 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume" pod "66ba75ef-f1f3-44c0-b296-8668a1c514af" (UID: "66ba75ef-f1f3-44c0-b296-8668a1c514af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:09.757339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.757323 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba75ef-f1f3-44c0-b296-8668a1c514af-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.757386 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.757342 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tv46\" (UniqueName: \"kubernetes.io/projected/252cf1fd-2366-4e63-bcd0-e1af768fec53-kube-api-access-2tv46\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.757386 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.757353 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252cf1fd-2366-4e63-bcd0-e1af768fec53-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:09.758843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.758817 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk" pod "66ba75ef-f1f3-44c0-b296-8668a1c514af" (UID: "66ba75ef-f1f3-44c0-b296-8668a1c514af"). InnerVolumeSpecName "kube-api-access-28zvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:09.878975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:09.864465 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zvk\" (UniqueName: \"kubernetes.io/projected/66ba75ef-f1f3-44c0-b296-8668a1c514af-kube-api-access-28zvk\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:10.277124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.277092 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" Oct 30 22:14:10.277124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.277115 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr" event={"ID":"252cf1fd-2366-4e63-bcd0-e1af768fec53","Type":"ContainerDied","Data":"1e2e38778085a85ee8af42adc827121ed16f63f10c3b2a7479f0cfd728a48314"} Oct 30 22:14:10.277661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.277177 5607 scope.go:122] "RemoveContainer" containerID="f82de597e0f7abb502673b969def000f9ba7b5d50849e6c8a8f76b723f1e118b" Oct 30 22:14:10.278667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.278644 5607 generic.go:356] "Generic (PLEG): container finished" podID="66ba75ef-f1f3-44c0-b296-8668a1c514af" containerID="d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981" exitCode=0 Oct 30 22:14:10.278728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.278682 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" Oct 30 22:14:10.278794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.278762 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" event={"ID":"66ba75ef-f1f3-44c0-b296-8668a1c514af","Type":"ContainerDied","Data":"d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981"} Oct 30 22:14:10.278865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.278809 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv" event={"ID":"66ba75ef-f1f3-44c0-b296-8668a1c514af","Type":"ContainerDied","Data":"b03957fb43d5cc288009ad95fc52c9c1739a4ac60b55994d02b600dc646e53c0"} Oct 30 22:14:10.280410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.280388 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" Oct 30 22:14:10.280410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.280397 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md" event={"ID":"a391201c-a4d6-454b-b829-060b7dc6d5c9","Type":"ContainerDied","Data":"e6308f4f83db74ebd46face7b36c4123f6a4b720a936e771a364270da2d5730d"} Oct 30 22:14:10.289864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.289842 5607 scope.go:122] "RemoveContainer" containerID="d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981" Oct 30 22:14:10.326588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.326563 5607 scope.go:122] "RemoveContainer" containerID="d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981" Oct 30 22:14:10.326994 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:10.326964 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981\": container with ID starting with d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981 not found: ID does not exist" containerID="d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981" Oct 30 22:14:10.327108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.327000 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981"} err="failed to get container status \"d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981\": rpc error: code = NotFound desc = could not find container \"d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981\": container with ID starting with d432cb433a6949d8285dff24a1c9d7a67ea2d0390d03ca2dbffdd93dfe3c6981 not found: ID does not exist" Oct 30 22:14:10.327108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.327022 5607 scope.go:122] "RemoveContainer" containerID="94a1a021fdbbddb7e044adff367f4e7fafac387bfeaa6d98af391d5a3e340812" Oct 30 22:14:10.385467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.385417 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p"] Oct 30 22:14:10.385728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.385683 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" containerID="cri-o://0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7" gracePeriod=30 Oct 30 22:14:10.388996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.388973 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n"] Oct 30 22:14:10.389181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.389161 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" containerID="cri-o://191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408" gracePeriod=30 Oct 30 22:14:10.435968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.435922 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp"] Oct 30 22:14:10.436228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.436204 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" containerID="cri-o://fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675" gracePeriod=30 Oct 30 22:14:10.438989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.438964 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7"] Oct 30 22:14:10.439323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.439297 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" containerID="cri-o://639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb" gracePeriod=30 Oct 30 22:14:10.451204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.451175 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md"] Oct 30 22:14:10.453738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.453718 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cf21208-84d9489ccd-9g8md"] Oct 30 22:14:10.588830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.588758 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr"] Oct 30 22:14:10.593293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.593266 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-872730-5874968845-brksr"] Oct 30 22:14:10.723069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.723037 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv"] Oct 30 22:14:10.726078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.726054 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-872730-6dd75c954c-9zvdv"] Oct 30 22:14:10.888778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.888246 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252cf1fd-2366-4e63-bcd0-e1af768fec53" path="/var/lib/kubelet/pods/252cf1fd-2366-4e63-bcd0-e1af768fec53/volumes" Oct 30 22:14:10.891296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.888937 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba75ef-f1f3-44c0-b296-8668a1c514af" path="/var/lib/kubelet/pods/66ba75ef-f1f3-44c0-b296-8668a1c514af/volumes" Oct 30 22:14:10.891296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:10.889564 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a391201c-a4d6-454b-b829-060b7dc6d5c9" path="/var/lib/kubelet/pods/a391201c-a4d6-454b-b829-060b7dc6d5c9/volumes" Oct 30 22:14:13.682648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:13.682612 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58"] Oct 30 22:14:13.683134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:13.682824 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" containerID="cri-o://59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be" gracePeriod=30 Oct 30 22:14:15.322444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.322364 5607 generic.go:356] "Generic (PLEG): container finished" podID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerID="baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252" exitCode=0 Oct 30 22:14:15.322444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.322417 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" event={"ID":"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0","Type":"ContainerDied","Data":"baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252"} Oct 30 22:14:15.322881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.322865 5607 scope.go:122] "RemoveContainer" containerID="baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252" Oct 30 22:14:15.324470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.324441 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:15.324539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.324510 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:15.324580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.324541 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:15.792213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.792180 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869405 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869466 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869565 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869611 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869718 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869791 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869882 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.869986 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.870025 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket\") pod \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\" (UID: \"7c6669dd-0cbe-4e86-9981-7774b973ba8c\") " Oct 30 22:14:15.870912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.870883 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.871607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.871010 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.871921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.871830 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:14:15.872480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.872436 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:15.874011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.873944 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.874346 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.874299 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.877196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.877160 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "kube-api-access-ccnjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:15.885128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.884824 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.885128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.884974 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:15.885128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.885075 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token" pod "7c6669dd-0cbe-4e86-9981-7774b973ba8c" (UID: "7c6669dd-0cbe-4e86-9981-7774b973ba8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:15.947833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.947793 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:14:15.955519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.955492 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:14:15.971312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971269 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971414 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971457 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971494 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971594 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971628 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971669 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971733 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.971862 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.971803 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert\") pod \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\" (UID: \"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5\") " Oct 30 22:14:15.972737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972683 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.972737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972716 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.972737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972732 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972747 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972761 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972776 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972789 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972802 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6669dd-0cbe-4e86-9981-7774b973ba8c-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972817 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccnjf\" (UniqueName: \"kubernetes.io/projected/7c6669dd-0cbe-4e86-9981-7774b973ba8c-kube-api-access-ccnjf\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.972685 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.973187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.973115 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.973658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.973314 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.973658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.973601 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:15.975725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.975415 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:15.975939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.975906 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:15.977758 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.977729 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:15.978189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.978161 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "kube-api-access-8trtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:15.978766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:15.978741 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy" pod "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" (UID: "dcb1adb5-7c81-4647-87a4-cd3f2820c2c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.074262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074227 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074268 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074311 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074347 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074371 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074393 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074418 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074472 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074510 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074551 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074605 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074621 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074647 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074647 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074710 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074772 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074797 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074837 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.074898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074855 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.075446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074931 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token\") pod \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\" (UID: \"76b4a034-13ec-4e36-9a62-17c4aa5008e0\") " Oct 30 22:14:16.075446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.074943 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.075446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.075030 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy\") pod \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\" (UID: \"3b0dc286-ed78-40f8-8f03-99132ba5aa74\") " Oct 30 22:14:16.075446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.075048 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.075446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.075302 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:16.076060 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076022 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076065 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076081 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076095 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076110 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076124 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076139 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076153 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076167 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076182 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trtk\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-kube-api-access-8trtk\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076198 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076212 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076227 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076240 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.076616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076254 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.077001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.076655 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:16.077076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.077017 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.077456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.077433 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.077625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.077603 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.077698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.077623 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:16.077880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.077860 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.078157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.078131 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:16.078270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.078220 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:16.080023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.079994 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "kube-api-access-5mfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:16.080023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.080015 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token" pod "76b4a034-13ec-4e36-9a62-17c4aa5008e0" (UID: "76b4a034-13ec-4e36-9a62-17c4aa5008e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:16.080356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.080334 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:16.080614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.080587 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd" pod "3b0dc286-ed78-40f8-8f03-99132ba5aa74" (UID: "3b0dc286-ed78-40f8-8f03-99132ba5aa74"). InnerVolumeSpecName "kube-api-access-nrsdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:16.178171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178139 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfbd\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-kube-api-access-5mfbd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178167 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178177 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178186 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178194 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178202 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178211 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178220 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178228 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178236 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrsdd\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-kube-api-access-nrsdd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178244 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0dc286-ed78-40f8-8f03-99132ba5aa74-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.178404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.178252 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4a034-13ec-4e36-9a62-17c4aa5008e0-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.332275 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.332173 5607 generic.go:356] "Generic (PLEG): container finished" podID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerID="639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb" exitCode=0 Oct 30 22:14:16.332275 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.332242 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" Oct 30 22:14:16.332866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.332271 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" event={"ID":"76b4a034-13ec-4e36-9a62-17c4aa5008e0","Type":"ContainerDied","Data":"639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb"} Oct 30 22:14:16.332866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.332333 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7" event={"ID":"76b4a034-13ec-4e36-9a62-17c4aa5008e0","Type":"ContainerDied","Data":"6258be61e3b174750516d772ea4dcfe7263bd99e138e1a524f0741740620a144"} Oct 30 22:14:16.332866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.332365 5607 scope.go:122] "RemoveContainer" containerID="639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb" Oct 30 22:14:16.334358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.334191 5607 generic.go:356] "Generic (PLEG): container finished" podID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerID="0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7" exitCode=0 Oct 30 22:14:16.334358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.334282 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" event={"ID":"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5","Type":"ContainerDied","Data":"0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7"} Oct 30 22:14:16.334358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.334324 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" event={"ID":"dcb1adb5-7c81-4647-87a4-cd3f2820c2c5","Type":"ContainerDied","Data":"2b8da06fd553e75700765a7fd0db62f58c7df44ae31013fe41e7e8ce3bca5797"} Oct 30 22:14:16.334720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.334390 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p" Oct 30 22:14:16.337026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.336804 5607 generic.go:356] "Generic (PLEG): container finished" podID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerID="191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408" exitCode=0 Oct 30 22:14:16.337026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.336884 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" event={"ID":"7c6669dd-0cbe-4e86-9981-7774b973ba8c","Type":"ContainerDied","Data":"191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408"} Oct 30 22:14:16.338351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.337773 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" event={"ID":"7c6669dd-0cbe-4e86-9981-7774b973ba8c","Type":"ContainerDied","Data":"02d62f11b9a4ba9246d392fcfd028b1def0730ab808573eeb6481fad119ed4fb"} Oct 30 22:14:16.339001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.338939 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n" Oct 30 22:14:16.339970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.339924 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" event={"ID":"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0","Type":"ContainerStarted","Data":"0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5"} Oct 30 22:14:16.346002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.345867 5607 generic.go:356] "Generic (PLEG): container finished" podID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerID="fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675" exitCode=0 Oct 30 22:14:16.346002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.345915 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" Oct 30 22:14:16.346002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.345939 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" event={"ID":"3b0dc286-ed78-40f8-8f03-99132ba5aa74","Type":"ContainerDied","Data":"fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675"} Oct 30 22:14:16.346202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.346011 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp" event={"ID":"3b0dc286-ed78-40f8-8f03-99132ba5aa74","Type":"ContainerDied","Data":"b97cb11c0988357ba8df5fb3fed41b5028ec3f5c4f89252fbf39f0ff3c3fdfb4"} Oct 30 22:14:16.348582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.348300 5607 scope.go:122] "RemoveContainer" containerID="639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb" Oct 30 22:14:16.348625 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:16.348594 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb\": container with ID starting with 639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb not found: ID does not exist" containerID="639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb" Oct 30 22:14:16.348661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.348624 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb"} err="failed to get container status \"639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb\": rpc error: code = NotFound desc = could not find container \"639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb\": container with ID starting with 639574de05177a8d2ef5e56c5039f05891f392c8d525b072d7a9f2ee4599b8eb not found: ID does not exist" Oct 30 22:14:16.348661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.348642 5607 scope.go:122] "RemoveContainer" containerID="0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7" Oct 30 22:14:16.359430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.359414 5607 scope.go:122] "RemoveContainer" containerID="0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7" Oct 30 22:14:16.359677 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:16.359662 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7\": container with ID starting with 0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7 not found: ID does not exist" containerID="0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7" Oct 30 22:14:16.359728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.359684 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7"} err="failed to get container status \"0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7\": rpc error: code = NotFound desc = could not find container \"0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7\": container with ID starting with 0f65625218fa580d1eeb851989e1368d03825b7e1b485449126804c0ff7438c7 not found: ID does not exist" Oct 30 22:14:16.359728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.359701 5607 scope.go:122] "RemoveContainer" containerID="191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408" Oct 30 22:14:16.369374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.369350 5607 scope.go:122] "RemoveContainer" containerID="191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408" Oct 30 22:14:16.369633 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:16.369613 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408\": container with ID starting with 191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408 not found: ID does not exist" containerID="191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408" Oct 30 22:14:16.369683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.369640 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408"} err="failed to get container status \"191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408\": rpc error: code = NotFound desc = could not find container \"191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408\": container with ID starting with 191968a3b9d027608c17faff1dc0f0984e8a7943da147fc4fa5af06d3e6d3408 not found: ID does not exist" Oct 30 22:14:16.369683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.369657 5607 scope.go:122] "RemoveContainer" containerID="fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675" Oct 30 22:14:16.378560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.378545 5607 scope.go:122] "RemoveContainer" containerID="fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675" Oct 30 22:14:16.378783 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:16.378766 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675\": container with ID starting with fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675 not found: ID does not exist" containerID="fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675" Oct 30 22:14:16.378825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.378789 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675"} err="failed to get container status \"fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675\": rpc error: code = NotFound desc = could not find container \"fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675\": container with ID starting with fcc0a22e9c66588382fcd51631c5ba2ada47759589e9338d920a719f13db4675 not found: ID does not exist" Oct 30 22:14:16.627944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.627874 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n"] Oct 30 22:14:16.632549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.632521 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l472n"] Oct 30 22:14:16.748652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.748619 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p"] Oct 30 22:14:16.753591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.753567 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-4qr6p"] Oct 30 22:14:16.756096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.756079 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:14:16.756238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.756221 5607 patch_prober.go:29] interesting pod/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:14:16.756281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.756256 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 22:14:16.873460 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.873425 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" path="/var/lib/kubelet/pods/7c6669dd-0cbe-4e86-9981-7774b973ba8c/volumes" Oct 30 22:14:16.873969 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.873940 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" path="/var/lib/kubelet/pods/dcb1adb5-7c81-4647-87a4-cd3f2820c2c5/volumes" Oct 30 22:14:16.874274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.874260 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp"] Oct 30 22:14:16.874355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.874294 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-2xklp"] Oct 30 22:14:16.991055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:16.991026 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7"] Oct 30 22:14:17.011434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:17.011395 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ndpw7"] Oct 30 22:14:17.756576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:17.756543 5607 patch_prober.go:29] interesting pod/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:14:17.757012 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:17.756607 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 22:14:17.972007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:17.971946 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:17.972178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:17.972020 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:18.756187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:18.756155 5607 patch_prober.go:29] interesting pod/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:14:18.756304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:18.756206 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 22:14:18.874990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:18.873183 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" path="/var/lib/kubelet/pods/3b0dc286-ed78-40f8-8f03-99132ba5aa74/volumes" Oct 30 22:14:18.874990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:18.874023 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" path="/var/lib/kubelet/pods/76b4a034-13ec-4e36-9a62-17c4aa5008e0/volumes" Oct 30 22:14:19.028970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.028925 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:14:19.140908 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.140866 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.140928 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.140997 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141049 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141181 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141214 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141327 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:19.141566 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141541 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.141657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141367 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141639 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.141657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141647 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141769 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy\") pod \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\" (UID: \"6575f77f-d61d-454d-9ed3-4c5fcf68c640\") " Oct 30 22:14:19.141881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141414 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.141881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.141819 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.142415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.142388 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.142647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.142420 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.142647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.142436 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.142647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.142450 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.142647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.142464 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.145071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.145046 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:19.145314 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.145292 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "kube-api-access-zdqsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.147562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.147543 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:19.147657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.147631 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token" pod "6575f77f-d61d-454d-9ed3-4c5fcf68c640" (UID: "6575f77f-d61d-454d-9ed3-4c5fcf68c640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:19.243702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.243620 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.243702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.243662 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.243702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.243680 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6575f77f-d61d-454d-9ed3-4c5fcf68c640-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.243702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.243697 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdqsd\" (UniqueName: \"kubernetes.io/projected/6575f77f-d61d-454d-9ed3-4c5fcf68c640-kube-api-access-zdqsd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:19.377312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.377273 5607 generic.go:356] "Generic (PLEG): container finished" podID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerID="59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be" exitCode=0 Oct 30 22:14:19.377554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.377534 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" event={"ID":"6575f77f-d61d-454d-9ed3-4c5fcf68c640","Type":"ContainerDied","Data":"59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be"} Oct 30 22:14:19.377609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.377595 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" event={"ID":"6575f77f-d61d-454d-9ed3-4c5fcf68c640","Type":"ContainerDied","Data":"331a0d8a6bf0332ea614777a56a8fca852c56500316a1b84444b86c611531951"} Oct 30 22:14:19.377645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.377628 5607 scope.go:122] "RemoveContainer" containerID="59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be" Oct 30 22:14:19.377798 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.377782 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58" Oct 30 22:14:19.428493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.428470 5607 scope.go:122] "RemoveContainer" containerID="59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be" Oct 30 22:14:19.428856 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:19.428835 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be\": container with ID starting with 59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be not found: ID does not exist" containerID="59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be" Oct 30 22:14:19.428907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.428867 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be"} err="failed to get container status \"59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be\": rpc error: code = NotFound desc = could not find container \"59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be\": container with ID starting with 59e6f2d3bec02bf299078f0b4d5fd89758edc5d93bdfdb9bfde3f61edc6414be not found: ID does not exist" Oct 30 22:14:19.553986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.553888 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58"] Oct 30 22:14:19.558486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.558464 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-njz58"] Oct 30 22:14:19.755660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.755625 5607 patch_prober.go:29] interesting pod/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:14:19.755851 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:19.755683 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 22:14:20.755844 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:20.755808 5607 patch_prober.go:29] interesting pod/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:14:20.756253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:20.755854 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 22:14:20.871303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:20.871261 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" path="/var/lib/kubelet/pods/6575f77f-d61d-454d-9ed3-4c5fcf68c640/volumes" Oct 30 22:14:21.756625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:21.756587 5607 patch_prober.go:29] interesting pod/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 22:14:21.757040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:21.756652 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 22:14:22.760724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:22.760693 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:14:22.761197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:22.760999 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:14:22.761690 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:22.761673 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:14:22.863904 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:22.863860 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:14:25.881105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881069 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh"] Oct 30 22:14:25.881827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881811 5607 cpu_manager.go:414] "containerMap: removing container" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881828 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881845 5607 cpu_manager.go:414] "containerMap: removing container" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881850 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881859 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881863 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881873 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a391201c-a4d6-454b-b829-060b7dc6d5c9" containerName="maas-api" Oct 30 22:14:25.881882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881877 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a391201c-a4d6-454b-b829-060b7dc6d5c9" containerName="maas-api" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881888 5607 cpu_manager.go:414] "containerMap: removing container" podUID="252cf1fd-2366-4e63-bcd0-e1af768fec53" containerName="payload-processing" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881894 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252cf1fd-2366-4e63-bcd0-e1af768fec53" containerName="payload-processing" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881910 5607 cpu_manager.go:414] "containerMap: removing container" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881916 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881923 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881927 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881936 5607 cpu_manager.go:414] "containerMap: removing container" podUID="66ba75ef-f1f3-44c0-b296-8668a1c514af" containerName="payload-pre-processing" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881940 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba75ef-f1f3-44c0-b296-8668a1c514af" containerName="payload-pre-processing" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881982 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.881987 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882078 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6575f77f-d61d-454d-9ed3-4c5fcf68c640" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882088 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b4a034-13ec-4e36-9a62-17c4aa5008e0" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882095 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="00710edf-10cd-4588-8fc4-68fdb78c78e9" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882100 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6669dd-0cbe-4e86-9981-7774b973ba8c" containerName="istio-proxy" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882106 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a391201c-a4d6-454b-b829-060b7dc6d5c9" containerName="maas-api" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882112 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="252cf1fd-2366-4e63-bcd0-e1af768fec53" containerName="payload-processing" Oct 30 22:14:25.882115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882120 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb1adb5-7c81-4647-87a4-cd3f2820c2c5" containerName="istio-proxy" Oct 30 22:14:25.882592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882125 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0dc286-ed78-40f8-8f03-99132ba5aa74" containerName="istio-proxy" Oct 30 22:14:25.882592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.882131 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba75ef-f1f3-44c0-b296-8668a1c514af" containerName="payload-pre-processing" Oct 30 22:14:25.885544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.885522 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:25.947675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.947641 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh"] Oct 30 22:14:25.963939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.963906 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:25.964411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:25.964388 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jchn\" (UniqueName: \"kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:26.050979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.042165 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4cf21208\"" Oct 30 22:14:26.072588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.051761 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4cf21208-dockercfg-r7d95\"" Oct 30 22:14:26.072588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.066057 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jchn\" (UniqueName: \"kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:26.072588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.066176 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:26.072588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.067260 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:26.091827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.091798 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655"] Oct 30 22:14:26.128259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.128226 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655"] Oct 30 22:14:26.128619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.128598 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.151556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.136131 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jchn\" (UniqueName: \"kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn\") pod \"payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:26.167427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.167396 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpx82\" (UniqueName: \"kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.167596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.167570 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.220229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.220197 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:26.268683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.268648 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpx82\" (UniqueName: \"kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.268848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.268776 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.269521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.269494 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.333792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.333752 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpx82\" (UniqueName: \"kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82\") pod \"payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.473133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.473104 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:26.618411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.617701 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh"] Oct 30 22:14:26.628981 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:14:26.625444 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3b65ac_f714_4bd1_9c7e_7feefc36316b.slice/crio-11f9a2cc8672c3d977bb1e3241d85f2af154358dd1ba3847d68d71f7f9fabe28 WatchSource:0}: Error finding container 11f9a2cc8672c3d977bb1e3241d85f2af154358dd1ba3847d68d71f7f9fabe28: Status 404 returned error can't find the container with id 11f9a2cc8672c3d977bb1e3241d85f2af154358dd1ba3847d68d71f7f9fabe28 Oct 30 22:14:26.877165 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:14:26.877135 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea75a11f_47ab_414b_b350_9e870be08c99.slice/crio-17ebc60301105380e7279f6ea1076334e8a45a3f23ac409c571aa16c9986f383 WatchSource:0}: Error finding container 17ebc60301105380e7279f6ea1076334e8a45a3f23ac409c571aa16c9986f383: Status 404 returned error can't find the container with id 17ebc60301105380e7279f6ea1076334e8a45a3f23ac409c571aa16c9986f383 Oct 30 22:14:26.879622 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:26.879598 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655"] Oct 30 22:14:27.448966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.448913 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" event={"ID":"ce3b65ac-f714-4bd1-9c7e-7feefc36316b","Type":"ContainerStarted","Data":"a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c"} Oct 30 22:14:27.449453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.448997 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" event={"ID":"ce3b65ac-f714-4bd1-9c7e-7feefc36316b","Type":"ContainerStarted","Data":"11f9a2cc8672c3d977bb1e3241d85f2af154358dd1ba3847d68d71f7f9fabe28"} Oct 30 22:14:27.449453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.449030 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:27.450814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.450789 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" event={"ID":"ea75a11f-47ab-414b-b350-9e870be08c99","Type":"ContainerStarted","Data":"4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2"} Oct 30 22:14:27.450989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.450832 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" event={"ID":"ea75a11f-47ab-414b-b350-9e870be08c99","Type":"ContainerStarted","Data":"17ebc60301105380e7279f6ea1076334e8a45a3f23ac409c571aa16c9986f383"} Oct 30 22:14:27.450989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.450871 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:27.578350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.578300 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" podStartSLOduration=2.578287817 podStartE2EDuration="2.578287817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:27.577232983 +0000 UTC m=+1646.977789097" watchObservedRunningTime="2026-10-30 22:14:27.578287817 +0000 UTC m=+1646.978843937" Oct 30 22:14:27.715126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:27.707122 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" podStartSLOduration=2.707100343 podStartE2EDuration="2.707100343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:27.696326378 +0000 UTC m=+1647.096882500" watchObservedRunningTime="2026-10-30 22:14:27.707100343 +0000 UTC m=+1647.107656465" Oct 30 22:14:33.496772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:33.496737 5607 generic.go:356] "Generic (PLEG): container finished" podID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerID="f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c" exitCode=0 Oct 30 22:14:33.497353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:33.496815 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" event={"ID":"551af49d-9c92-48ed-9a8f-831c634c31bd","Type":"ContainerDied","Data":"f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c"} Oct 30 22:14:33.497444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:33.497351 5607 scope.go:122] "RemoveContainer" containerID="f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c" Oct 30 22:14:33.498821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:33.498788 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:33.498917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:33.498862 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:33.498976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:33.498919 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:34.533242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:34.533192 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" event={"ID":"551af49d-9c92-48ed-9a8f-831c634c31bd","Type":"ContainerStarted","Data":"eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5"} Oct 30 22:14:35.113986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:35.113163 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:14:35.113986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:35.113605 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:14:35.113986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:35.113682 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 22:14:36.111978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:36.109218 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:14:36.111978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:36.109288 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 22:14:37.117978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:37.117248 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:14:37.117978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:37.117304 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 22:14:38.115978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:38.112246 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:14:38.115978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:38.112313 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 22:14:38.461613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:38.461586 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:14:38.462018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:38.461651 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:14:39.115271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:39.115238 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:14:39.115570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:39.115543 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 22:14:40.112597 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:40.112566 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:14:40.113216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:40.112996 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:14:40.113568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:40.113550 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:14:40.614280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:40.614249 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8"] Oct 30 22:14:40.729524 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:40.729495 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd"] Oct 30 22:14:43.715356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:43.715325 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5"] Oct 30 22:14:43.715783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:43.715527 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" containerID="cri-o://8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee" gracePeriod=30 Oct 30 22:14:45.614568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:45.614526 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" podUID="d6605229-d3aa-4a76-861c-4d429b9b4991" containerName="payload-processing" containerID="cri-o://7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8" gracePeriod=25 Oct 30 22:14:45.729840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:45.729808 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" podUID="b58d1533-83ed-484d-9f32-5c4b54a10dea" containerName="payload-pre-processing" containerID="cri-o://21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07" gracePeriod=25 Oct 30 22:14:45.922438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:45.922394 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:14:46.002582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.002559 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:14:46.025697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.025675 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume\") pod \"d6605229-d3aa-4a76-861c-4d429b9b4991\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " Oct 30 22:14:46.025841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.025824 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq\" (UniqueName: \"kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq\") pod \"d6605229-d3aa-4a76-861c-4d429b9b4991\" (UID: \"d6605229-d3aa-4a76-861c-4d429b9b4991\") " Oct 30 22:14:46.026092 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.026074 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume" pod "d6605229-d3aa-4a76-861c-4d429b9b4991" (UID: "d6605229-d3aa-4a76-861c-4d429b9b4991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:46.026360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.026341 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6605229-d3aa-4a76-861c-4d429b9b4991-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:46.027947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.027927 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq" pod "d6605229-d3aa-4a76-861c-4d429b9b4991" (UID: "d6605229-d3aa-4a76-861c-4d429b9b4991"). InnerVolumeSpecName "kube-api-access-pl4hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:46.126925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.126892 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume\") pod \"b58d1533-83ed-484d-9f32-5c4b54a10dea\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " Oct 30 22:14:46.127121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.127009 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4\" (UniqueName: \"kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4\") pod \"b58d1533-83ed-484d-9f32-5c4b54a10dea\" (UID: \"b58d1533-83ed-484d-9f32-5c4b54a10dea\") " Oct 30 22:14:46.127344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.127321 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume" pod "b58d1533-83ed-484d-9f32-5c4b54a10dea" (UID: "b58d1533-83ed-484d-9f32-5c4b54a10dea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:46.127580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.127565 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl4hq\" (UniqueName: \"kubernetes.io/projected/d6605229-d3aa-4a76-861c-4d429b9b4991-kube-api-access-pl4hq\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:46.127624 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.127583 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b58d1533-83ed-484d-9f32-5c4b54a10dea-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:46.129191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.129173 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4" pod "b58d1533-83ed-484d-9f32-5c4b54a10dea" (UID: "b58d1533-83ed-484d-9f32-5c4b54a10dea"). InnerVolumeSpecName "kube-api-access-54rl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:46.229063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.229032 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rl4\" (UniqueName: \"kubernetes.io/projected/b58d1533-83ed-484d-9f32-5c4b54a10dea-kube-api-access-54rl4\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:46.635497 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.635462 5607 generic.go:356] "Generic (PLEG): container finished" podID="d6605229-d3aa-4a76-861c-4d429b9b4991" containerID="7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8" exitCode=0 Oct 30 22:14:46.636051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.635509 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" Oct 30 22:14:46.636051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.635540 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" event={"ID":"d6605229-d3aa-4a76-861c-4d429b9b4991","Type":"ContainerDied","Data":"7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8"} Oct 30 22:14:46.636051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.635619 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8" event={"ID":"d6605229-d3aa-4a76-861c-4d429b9b4991","Type":"ContainerDied","Data":"84a0feec7a3f0e4a6e3cda77b07631e08e88f4186e2f2b2f3477eac725365fcd"} Oct 30 22:14:46.636051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.635646 5607 scope.go:122] "RemoveContainer" containerID="7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8" Oct 30 22:14:46.637345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.637318 5607 generic.go:356] "Generic (PLEG): container finished" podID="b58d1533-83ed-484d-9f32-5c4b54a10dea" containerID="21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07" exitCode=0 Oct 30 22:14:46.637458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.637381 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" Oct 30 22:14:46.637458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.637405 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" event={"ID":"b58d1533-83ed-484d-9f32-5c4b54a10dea","Type":"ContainerDied","Data":"21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07"} Oct 30 22:14:46.637574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.637488 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd" event={"ID":"b58d1533-83ed-484d-9f32-5c4b54a10dea","Type":"ContainerDied","Data":"ae500a83808895e50323674919c2ed199c3038e6e22e10b31c7ce9aedef956c4"} Oct 30 22:14:46.648529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.648511 5607 scope.go:122] "RemoveContainer" containerID="7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8" Oct 30 22:14:46.648877 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:46.648857 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8\": container with ID starting with 7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8 not found: ID does not exist" containerID="7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8" Oct 30 22:14:46.648947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.648883 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8"} err="failed to get container status \"7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8\": rpc error: code = NotFound desc = could not find container \"7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8\": container with ID starting with 7b7c0261a23396221a78a2d9ce112ffcf08f1d063227969a747acb5d360aa1d8 not found: ID does not exist" Oct 30 22:14:46.648947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.648900 5607 scope.go:122] "RemoveContainer" containerID="21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07" Oct 30 22:14:46.657881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.657866 5607 scope.go:122] "RemoveContainer" containerID="21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07" Oct 30 22:14:46.658171 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:46.658150 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07\": container with ID starting with 21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07 not found: ID does not exist" containerID="21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07" Oct 30 22:14:46.658244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.658179 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07"} err="failed to get container status \"21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07\": rpc error: code = NotFound desc = could not find container \"21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07\": container with ID starting with 21600698c95028ddea2da4b5f4af5b83e67cd3ac6ada88fd97735eff24d54e07 not found: ID does not exist" Oct 30 22:14:46.808155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.808123 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8"] Oct 30 22:14:46.818968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.818929 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-161111-786d9b8465-52bs8"] Oct 30 22:14:46.871170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.871141 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6605229-d3aa-4a76-861c-4d429b9b4991" path="/var/lib/kubelet/pods/d6605229-d3aa-4a76-861c-4d429b9b4991/volumes" Oct 30 22:14:46.936478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.936452 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd"] Oct 30 22:14:46.947399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:46.947366 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-161111-5cf6885b99-b2bcd"] Oct 30 22:14:47.260052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:47.259965 5607 patch_prober.go:29] interesting pod/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:14:47.260052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:47.260036 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:14:48.872420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:48.872387 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b58d1533-83ed-484d-9f32-5c4b54a10dea" path="/var/lib/kubelet/pods/b58d1533-83ed-484d-9f32-5c4b54a10dea/volumes" Oct 30 22:14:48.990164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:48.990100 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:14:49.070876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.070847 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.070895 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.070923 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071108 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071145 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071204 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071251 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071282 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071309 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:49.071459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071327 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:49.071459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071382 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket\") pod \"40058b19-0e7f-4b5d-a417-e296ca5e993d\" (UID: \"40058b19-0e7f-4b5d-a417-e296ca5e993d\") " Oct 30 22:14:49.071623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071474 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:49.071623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071613 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:49.071786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.071763 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:49.072085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.072066 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.072171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.072089 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.072171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.072103 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.072171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.072115 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40058b19-0e7f-4b5d-a417-e296ca5e993d-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.072171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.072128 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.073452 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.073427 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:14:49.073546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.073474 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "kube-api-access-vnn49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:49.073967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.073930 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:49.073967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.073938 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo" pod "40058b19-0e7f-4b5d-a417-e296ca5e993d" (UID: "40058b19-0e7f-4b5d-a417-e296ca5e993d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:14:49.173970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.173932 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.174140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.173976 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.174140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.173994 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.174140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.174004 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnn49\" (UniqueName: \"kubernetes.io/projected/40058b19-0e7f-4b5d-a417-e296ca5e993d-kube-api-access-vnn49\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:14:49.664026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.662609 5607 generic.go:356] "Generic (PLEG): container finished" podID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerID="8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee" exitCode=0 Oct 30 22:14:49.664026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.662898 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" event={"ID":"40058b19-0e7f-4b5d-a417-e296ca5e993d","Type":"ContainerDied","Data":"8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee"} Oct 30 22:14:49.664026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.663020 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" event={"ID":"40058b19-0e7f-4b5d-a417-e296ca5e993d","Type":"ContainerDied","Data":"bc594a929cc1d176d434bda2517199cf3ac804d900da4e96024f045451671a75"} Oct 30 22:14:49.664026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.663060 5607 scope.go:122] "RemoveContainer" containerID="8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee" Oct 30 22:14:49.664026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.663243 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5" Oct 30 22:14:49.682665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.682645 5607 scope.go:122] "RemoveContainer" containerID="6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a" Oct 30 22:14:49.693436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.693420 5607 scope.go:122] "RemoveContainer" containerID="8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee" Oct 30 22:14:49.693703 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:49.693683 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee\": container with ID starting with 8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee not found: ID does not exist" containerID="8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee" Oct 30 22:14:49.693770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.693715 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee"} err="failed to get container status \"8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee\": rpc error: code = NotFound desc = could not find container \"8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee\": container with ID starting with 8b7aefb5ace56b533f65c49c5bc793e6d723e8b89bf37b27034b31e2e37ebeee not found: ID does not exist" Oct 30 22:14:49.693770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.693733 5607 scope.go:122] "RemoveContainer" containerID="6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a" Oct 30 22:14:49.694036 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:49.694017 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a\": container with ID starting with 6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a not found: ID does not exist" containerID="6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a" Oct 30 22:14:49.694083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.694044 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a"} err="failed to get container status \"6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a\": rpc error: code = NotFound desc = could not find container \"6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a\": container with ID starting with 6e23fcee8fb6283662ac2d5118e281d5a13e2303390116f1623a560d3d5d5b7a not found: ID does not exist" Oct 30 22:14:49.877214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.877186 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5"] Oct 30 22:14:49.928822 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:49.928725 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dbe0a7-openshift-default-668db66569-ht9t5"] Oct 30 22:14:50.872210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:50.872181 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" path="/var/lib/kubelet/pods/40058b19-0e7f-4b5d-a417-e296ca5e993d/volumes" Oct 30 22:14:52.865267 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:14:52.865228 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:14:53.913647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.913608 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648"] Oct 30 22:14:53.914450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914431 5607 cpu_manager.go:414] "containerMap: removing container" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" Oct 30 22:14:53.914523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914452 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" Oct 30 22:14:53.914523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914463 5607 cpu_manager.go:414] "containerMap: removing container" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" Oct 30 22:14:53.914523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914469 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" Oct 30 22:14:53.914523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914493 5607 cpu_manager.go:414] "containerMap: removing container" podUID="b58d1533-83ed-484d-9f32-5c4b54a10dea" containerName="payload-pre-processing" Oct 30 22:14:53.914523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914498 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58d1533-83ed-484d-9f32-5c4b54a10dea" containerName="payload-pre-processing" Oct 30 22:14:53.914679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914533 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d6605229-d3aa-4a76-861c-4d429b9b4991" containerName="payload-processing" Oct 30 22:14:53.914679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914538 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6605229-d3aa-4a76-861c-4d429b9b4991" containerName="payload-processing" Oct 30 22:14:53.914679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914629 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6605229-d3aa-4a76-861c-4d429b9b4991" containerName="payload-processing" Oct 30 22:14:53.914679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914639 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" Oct 30 22:14:53.914679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914644 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="40058b19-0e7f-4b5d-a417-e296ca5e993d" containerName="istio-proxy" Oct 30 22:14:53.914679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.914655 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58d1533-83ed-484d-9f32-5c4b54a10dea" containerName="payload-pre-processing" Oct 30 22:14:53.918210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.918193 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:53.973191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.973147 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ad7da0\"" Oct 30 22:14:53.977457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:53.977429 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648"] Oct 30 22:14:54.043976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.043530 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ad7da0-dockercfg-k2gnj\"" Oct 30 22:14:54.067982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.047073 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.067982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.047524 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnn9\" (UniqueName: \"kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.067982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.065942 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q"] Oct 30 22:14:54.074565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.071408 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.101977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.101383 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q"] Oct 30 22:14:54.148997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.148940 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.149168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.149105 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnn9\" (UniqueName: \"kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.149710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.149684 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.212470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.212434 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnn9\" (UniqueName: \"kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9\") pod \"payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.232449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.232426 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.250632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.250608 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvjj\" (UniqueName: \"kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj\") pod \"payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.250816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.250795 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.352135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.352091 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.352356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.352236 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvjj\" (UniqueName: \"kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj\") pod \"payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.353585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.353561 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.415198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.415167 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvjj\" (UniqueName: \"kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj\") pod \"payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.421053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.421032 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:54.610732 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:14:54.610702 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25cea18_75f3_443d_957d_7d32ff2a1322.slice/crio-18f0f041938f82b416eb7615ed20cb8f3b4afa3b9534b5e5647dfd0a2d20cdeb WatchSource:0}: Error finding container 18f0f041938f82b416eb7615ed20cb8f3b4afa3b9534b5e5647dfd0a2d20cdeb: Status 404 returned error can't find the container with id 18f0f041938f82b416eb7615ed20cb8f3b4afa3b9534b5e5647dfd0a2d20cdeb Oct 30 22:14:54.614253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.614229 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648"] Oct 30 22:14:54.704665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.704627 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" event={"ID":"f25cea18-75f3-443d-957d-7d32ff2a1322","Type":"ContainerStarted","Data":"db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113"} Oct 30 22:14:54.704853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.704700 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:14:54.704853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.704724 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" event={"ID":"f25cea18-75f3-443d-957d-7d32ff2a1322","Type":"ContainerStarted","Data":"18f0f041938f82b416eb7615ed20cb8f3b4afa3b9534b5e5647dfd0a2d20cdeb"} Oct 30 22:14:54.797114 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:14:54.797084 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f090d42_ab12_4e84_af58_5370016c2733.slice/crio-6f26dd7725840a80431a213041290fa154bffbec12b414b3c517c952dad48e54 WatchSource:0}: Error finding container 6f26dd7725840a80431a213041290fa154bffbec12b414b3c517c952dad48e54: Status 404 returned error can't find the container with id 6f26dd7725840a80431a213041290fa154bffbec12b414b3c517c952dad48e54 Oct 30 22:14:54.799966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.799922 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q"] Oct 30 22:14:54.828384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:54.828327 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" podStartSLOduration=1.8283105 podStartE2EDuration="1.8283105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:54.827512995 +0000 UTC m=+1674.228069119" watchObservedRunningTime="2026-10-30 22:14:54.8283105 +0000 UTC m=+1674.228866622" Oct 30 22:14:55.713941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:55.713904 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" event={"ID":"2f090d42-ab12-4e84-af58-5370016c2733","Type":"ContainerStarted","Data":"39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba"} Oct 30 22:14:55.714399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:55.714006 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" event={"ID":"2f090d42-ab12-4e84-af58-5370016c2733","Type":"ContainerStarted","Data":"6f26dd7725840a80431a213041290fa154bffbec12b414b3c517c952dad48e54"} Oct 30 22:14:55.835220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:55.835162 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" podStartSLOduration=2.835146109 podStartE2EDuration="2.835146109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:55.83356898 +0000 UTC m=+1675.234125102" watchObservedRunningTime="2026-10-30 22:14:55.835146109 +0000 UTC m=+1675.235702229" Oct 30 22:14:56.721263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:56.721230 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:14:59.947052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:14:59.947021 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf"] Oct 30 22:15:00.197835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.197755 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw"] Oct 30 22:15:00.203588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.203564 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.208547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.208026 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql"] Oct 30 22:15:00.212709 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.212691 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.247761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.247734 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q"] Oct 30 22:15:00.251837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.251821 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz"] Oct 30 22:15:00.251968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.251934 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.255711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.255695 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.271811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.271791 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw"] Oct 30 22:15:00.289070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.289046 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d"] Oct 30 22:15:00.294548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.294530 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.294678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.294530 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql"] Oct 30 22:15:00.314482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.314455 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q"] Oct 30 22:15:00.336354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.331834 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-b9fql\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.336354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.331987 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dw4d\" (UniqueName: \"kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.336354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.332153 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfcq2\" (UniqueName: \"kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2\") pod \"maas-api-key-cleanup-29889975-b9fql\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.336354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.332282 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.358413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.358384 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:15:00.369132 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.369105 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d"] Oct 30 22:15:00.370430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.370406 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz"] Oct 30 22:15:00.433274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433246 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.433425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433284 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57qh\" (UniqueName: \"kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh\") pod \"maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.433425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433364 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmmr\" (UniqueName: \"kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr\") pod \"maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.433425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433411 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dw4d\" (UniqueName: \"kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.433550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433488 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.433585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433550 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfcq2\" (UniqueName: \"kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2\") pod \"maas-api-key-cleanup-29889975-b9fql\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.433633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433616 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.433728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433704 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.433848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433833 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9d7g\" (UniqueName: \"kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g\") pod \"maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.433904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.433890 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-b9fql\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.434334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.434319 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.434470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.434453 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-b9fql\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.497269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.497199 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dw4d\" (UniqueName: \"kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.497397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.497336 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfcq2\" (UniqueName: \"kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2\") pod \"maas-api-key-cleanup-29889975-b9fql\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.523446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.523421 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:00.529725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.529696 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:00.535627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.535604 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.535707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.535652 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57qh\" (UniqueName: \"kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh\") pod \"maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.535752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.535718 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmmr\" (UniqueName: \"kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr\") pod \"maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.535832 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.535817 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.535920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.535898 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.536409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.536375 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.536487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.536471 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.536651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.536633 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.536825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.536808 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9d7g\" (UniqueName: \"kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g\") pod \"maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.607602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.607574 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9d7g\" (UniqueName: \"kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g\") pod \"maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.632204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.632157 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmmr\" (UniqueName: \"kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr\") pod \"maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.632342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.632164 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57qh\" (UniqueName: \"kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh\") pod \"maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.866859 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.866787 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:00.875726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.875704 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:00.909050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.909023 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:00.927817 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:00.927792 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd618f1d1_4446_4931_8c4d_5886122097d2.slice/crio-ee4bab1f434ba81f2fcccad5518ebf9e80d5d02d5785c57747757f203dd5deeb WatchSource:0}: Error finding container ee4bab1f434ba81f2fcccad5518ebf9e80d5d02d5785c57747757f203dd5deeb: Status 404 returned error can't find the container with id ee4bab1f434ba81f2fcccad5518ebf9e80d5d02d5785c57747757f203dd5deeb Oct 30 22:15:00.927930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:00.927899 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw"] Oct 30 22:15:01.005453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.005427 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql"] Oct 30 22:15:01.014165 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:01.014130 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819b5d6f_8cbf_46f5_86cf_6cb193337970.slice/crio-87662f74498156b9c0e759dd9641cbdaa74de394bd29a3752c6d4ae3ee15771b WatchSource:0}: Error finding container 87662f74498156b9c0e759dd9641cbdaa74de394bd29a3752c6d4ae3ee15771b: Status 404 returned error can't find the container with id 87662f74498156b9c0e759dd9641cbdaa74de394bd29a3752c6d4ae3ee15771b Oct 30 22:15:01.251569 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:01.251529 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb8d3dc_d327_46ef_a13a_7a443823ed14.slice/crio-e95749012ce4b6b0c9ff3e42472311e104a13a2d7a1205a9ad4d4b6af19c11d8 WatchSource:0}: Error finding container e95749012ce4b6b0c9ff3e42472311e104a13a2d7a1205a9ad4d4b6af19c11d8: Status 404 returned error can't find the container with id e95749012ce4b6b0c9ff3e42472311e104a13a2d7a1205a9ad4d4b6af19c11d8 Oct 30 22:15:01.255389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.255368 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q"] Oct 30 22:15:01.267518 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:01.267490 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc03183_d218_450b_9e75_8acc338a1d98.slice/crio-f4409e5cb06eb7181fb71563072f9d42044ba58e292ceef2073172209f890da6 WatchSource:0}: Error finding container f4409e5cb06eb7181fb71563072f9d42044ba58e292ceef2073172209f890da6: Status 404 returned error can't find the container with id f4409e5cb06eb7181fb71563072f9d42044ba58e292ceef2073172209f890da6 Oct 30 22:15:01.271361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.271339 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz"] Oct 30 22:15:01.377074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.377052 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d"] Oct 30 22:15:01.400230 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:01.400201 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225f43ca_e55f_49fe_822c_baac58e23d4c.slice/crio-09bd192916d18af80672c9c8d4474293e44b8ad311ac0377705bce6c86b273aa WatchSource:0}: Error finding container 09bd192916d18af80672c9c8d4474293e44b8ad311ac0377705bce6c86b273aa: Status 404 returned error can't find the container with id 09bd192916d18af80672c9c8d4474293e44b8ad311ac0377705bce6c86b273aa Oct 30 22:15:01.577335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.577300 5607 scope.go:122] "RemoveContainer" containerID="6bfa92c0dacf0faae5bf89691a12a272bbede97a133d2fceea620faf91a3bad2" Oct 30 22:15:01.589430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.589407 5607 scope.go:122] "RemoveContainer" containerID="a4834c4b4eba86691999f26611b5ebb2f7dfe579e154ba6d3d6deaefe0326976" Oct 30 22:15:01.600427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.600405 5607 scope.go:122] "RemoveContainer" containerID="61312468e2bf49594c79bd5143620ded1a0a377a8532b570e9b4be858075276b" Oct 30 22:15:01.612141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.612060 5607 scope.go:122] "RemoveContainer" containerID="3770323ebffff9362b09a562223fa8f5659dded1e8bb6a9dd21daae81ee9402e" Oct 30 22:15:01.767770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.767030 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerStarted","Data":"cd7c33743c2152a3e2d9db1d3c06874af244b45557d787ca776f8ab658983407"} Oct 30 22:15:01.767770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.767148 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerStarted","Data":"ee4bab1f434ba81f2fcccad5518ebf9e80d5d02d5785c57747757f203dd5deeb"} Oct 30 22:15:01.769780 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.769740 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" event={"ID":"fbc03183-d218-450b-9e75-8acc338a1d98","Type":"ContainerStarted","Data":"34908a1fbf4fa8f0db3d2fb344788f207bac015d3fbf587a0865b9609579d48e"} Oct 30 22:15:01.769971 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.769798 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" event={"ID":"fbc03183-d218-450b-9e75-8acc338a1d98","Type":"ContainerStarted","Data":"f4409e5cb06eb7181fb71563072f9d42044ba58e292ceef2073172209f890da6"} Oct 30 22:15:01.771930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.771905 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerStarted","Data":"568759dec20c4d056103e766e3bb15c36a8934d363e7bdbef3326e9e763d0c38"} Oct 30 22:15:01.772118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.771946 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerStarted","Data":"87662f74498156b9c0e759dd9641cbdaa74de394bd29a3752c6d4ae3ee15771b"} Oct 30 22:15:01.773389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.773360 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" event={"ID":"225f43ca-e55f-49fe-822c-baac58e23d4c","Type":"ContainerStarted","Data":"09bd192916d18af80672c9c8d4474293e44b8ad311ac0377705bce6c86b273aa"} Oct 30 22:15:01.774792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.774770 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" event={"ID":"fdb8d3dc-d327-46ef-a13a-7a443823ed14","Type":"ContainerStarted","Data":"e95749012ce4b6b0c9ff3e42472311e104a13a2d7a1205a9ad4d4b6af19c11d8"} Oct 30 22:15:01.898130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:01.898065 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" podStartSLOduration=1.898042255 podStartE2EDuration="1.898042255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.893318799 +0000 UTC m=+1681.293874921" watchObservedRunningTime="2026-10-30 22:15:01.898042255 +0000 UTC m=+1681.298598376" Oct 30 22:15:02.030469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.030421 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" podStartSLOduration=2.0304082980000002 podStartE2EDuration="2.030408298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.028138275 +0000 UTC m=+1681.428694399" watchObservedRunningTime="2026-10-30 22:15:02.030408298 +0000 UTC m=+1681.430964419" Oct 30 22:15:02.148889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.148847 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" podStartSLOduration=2.148832721 podStartE2EDuration="2.148832721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.145716419 +0000 UTC m=+1681.546272540" watchObservedRunningTime="2026-10-30 22:15:02.148832721 +0000 UTC m=+1681.549388862" Oct 30 22:15:02.783592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.783559 5607 generic.go:356] "Generic (PLEG): container finished" podID="fbc03183-d218-450b-9e75-8acc338a1d98" containerID="34908a1fbf4fa8f0db3d2fb344788f207bac015d3fbf587a0865b9609579d48e" exitCode=60 Oct 30 22:15:02.783780 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.783642 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" event={"ID":"fbc03183-d218-450b-9e75-8acc338a1d98","Type":"ContainerDied","Data":"34908a1fbf4fa8f0db3d2fb344788f207bac015d3fbf587a0865b9609579d48e"} Oct 30 22:15:02.783875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.783858 5607 scope.go:122] "RemoveContainer" containerID="34908a1fbf4fa8f0db3d2fb344788f207bac015d3fbf587a0865b9609579d48e" Oct 30 22:15:02.785424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.785397 5607 generic.go:356] "Generic (PLEG): container finished" podID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerID="568759dec20c4d056103e766e3bb15c36a8934d363e7bdbef3326e9e763d0c38" exitCode=60 Oct 30 22:15:02.785546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.785476 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerDied","Data":"568759dec20c4d056103e766e3bb15c36a8934d363e7bdbef3326e9e763d0c38"} Oct 30 22:15:02.785683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.785651 5607 scope.go:122] "RemoveContainer" containerID="568759dec20c4d056103e766e3bb15c36a8934d363e7bdbef3326e9e763d0c38" Oct 30 22:15:02.789511 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.789488 5607 generic.go:356] "Generic (PLEG): container finished" podID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerID="eda7efa584194ac07743410bc6a9288fcc1a65777ea7f8b7042a71036b6a5eb6" exitCode=60 Oct 30 22:15:02.789586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.789554 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" event={"ID":"225f43ca-e55f-49fe-822c-baac58e23d4c","Type":"ContainerDied","Data":"eda7efa584194ac07743410bc6a9288fcc1a65777ea7f8b7042a71036b6a5eb6"} Oct 30 22:15:02.789797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.789783 5607 scope.go:122] "RemoveContainer" containerID="eda7efa584194ac07743410bc6a9288fcc1a65777ea7f8b7042a71036b6a5eb6" Oct 30 22:15:02.791428 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.791405 5607 generic.go:356] "Generic (PLEG): container finished" podID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerID="7954a688194229da8a5c1d373d5511c1b5bc3f62a56f7f6894edaba4180d200a" exitCode=60 Oct 30 22:15:02.791540 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.791476 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" event={"ID":"fdb8d3dc-d327-46ef-a13a-7a443823ed14","Type":"ContainerDied","Data":"7954a688194229da8a5c1d373d5511c1b5bc3f62a56f7f6894edaba4180d200a"} Oct 30 22:15:02.791728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.791712 5607 scope.go:122] "RemoveContainer" containerID="7954a688194229da8a5c1d373d5511c1b5bc3f62a56f7f6894edaba4180d200a" Oct 30 22:15:02.793138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.793119 5607 generic.go:356] "Generic (PLEG): container finished" podID="d618f1d1-4446-4931-8c4d-5886122097d2" containerID="cd7c33743c2152a3e2d9db1d3c06874af244b45557d787ca776f8ab658983407" exitCode=60 Oct 30 22:15:02.793262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.793197 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerDied","Data":"cd7c33743c2152a3e2d9db1d3c06874af244b45557d787ca776f8ab658983407"} Oct 30 22:15:02.793394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:02.793378 5607 scope.go:122] "RemoveContainer" containerID="cd7c33743c2152a3e2d9db1d3c06874af244b45557d787ca776f8ab658983407" Oct 30 22:15:03.807827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.807780 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" event={"ID":"225f43ca-e55f-49fe-822c-baac58e23d4c","Type":"ContainerStarted","Data":"4a0147ca60c590f22ddc4953853202db1a57190062083224a1e98cc878e48ead"} Oct 30 22:15:03.810010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.809985 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" event={"ID":"fdb8d3dc-d327-46ef-a13a-7a443823ed14","Type":"ContainerStarted","Data":"5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc"} Oct 30 22:15:03.812326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.812292 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerStarted","Data":"a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657"} Oct 30 22:15:03.814250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.814226 5607 generic.go:356] "Generic (PLEG): container finished" podID="fbc03183-d218-450b-9e75-8acc338a1d98" containerID="7ea4dd3ae44d0630da0a5da9072b107b8a1f86cb311c8af74997489ba121cd2c" exitCode=60 Oct 30 22:15:03.814368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.814302 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" event={"ID":"fbc03183-d218-450b-9e75-8acc338a1d98","Type":"ContainerDied","Data":"7ea4dd3ae44d0630da0a5da9072b107b8a1f86cb311c8af74997489ba121cd2c"} Oct 30 22:15:03.814368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.814356 5607 scope.go:122] "RemoveContainer" containerID="34908a1fbf4fa8f0db3d2fb344788f207bac015d3fbf587a0865b9609579d48e" Oct 30 22:15:03.814616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.814596 5607 scope.go:122] "RemoveContainer" containerID="7ea4dd3ae44d0630da0a5da9072b107b8a1f86cb311c8af74997489ba121cd2c" Oct 30 22:15:03.815199 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:03.814939 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz_odh-ai-gateway-infra(fbc03183-d218-450b-9e75-8acc338a1d98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" Oct 30 22:15:03.816473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.816453 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerStarted","Data":"148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9"} Oct 30 22:15:03.930503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:03.930456 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" podStartSLOduration=3.930440581 podStartE2EDuration="3.930440581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.928600414 +0000 UTC m=+1683.329156536" watchObservedRunningTime="2026-10-30 22:15:03.930440581 +0000 UTC m=+1683.330997022" Oct 30 22:15:04.048388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.048349 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" podStartSLOduration=4.048337474 podStartE2EDuration="4.048337474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.047597696 +0000 UTC m=+1683.448153818" watchObservedRunningTime="2026-10-30 22:15:04.048337474 +0000 UTC m=+1683.448893595" Oct 30 22:15:04.831171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.831130 5607 generic.go:356] "Generic (PLEG): container finished" podID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerID="4a0147ca60c590f22ddc4953853202db1a57190062083224a1e98cc878e48ead" exitCode=60 Oct 30 22:15:04.831648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.831220 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" event={"ID":"225f43ca-e55f-49fe-822c-baac58e23d4c","Type":"ContainerDied","Data":"4a0147ca60c590f22ddc4953853202db1a57190062083224a1e98cc878e48ead"} Oct 30 22:15:04.831648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.831307 5607 scope.go:122] "RemoveContainer" containerID="eda7efa584194ac07743410bc6a9288fcc1a65777ea7f8b7042a71036b6a5eb6" Oct 30 22:15:04.831741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.831703 5607 scope.go:122] "RemoveContainer" containerID="4a0147ca60c590f22ddc4953853202db1a57190062083224a1e98cc878e48ead" Oct 30 22:15:04.832151 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:04.832121 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d_odh-ai-gateway-infra(225f43ca-e55f-49fe-822c-baac58e23d4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" Oct 30 22:15:04.833549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.833528 5607 generic.go:356] "Generic (PLEG): container finished" podID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerID="5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc" exitCode=60 Oct 30 22:15:04.833680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.833619 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" event={"ID":"fdb8d3dc-d327-46ef-a13a-7a443823ed14","Type":"ContainerDied","Data":"5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc"} Oct 30 22:15:04.833904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.833886 5607 scope.go:122] "RemoveContainer" containerID="5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc" Oct 30 22:15:04.834249 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:04.834228 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q_odh-ai-gateway-infra(fdb8d3dc-d327-46ef-a13a-7a443823ed14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" Oct 30 22:15:04.835674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.835651 5607 generic.go:356] "Generic (PLEG): container finished" podID="d618f1d1-4446-4931-8c4d-5886122097d2" containerID="a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657" exitCode=60 Oct 30 22:15:04.835791 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.835766 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerDied","Data":"a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657"} Oct 30 22:15:04.835995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.835978 5607 scope.go:122] "RemoveContainer" containerID="a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657" Oct 30 22:15:04.836289 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:04.836270 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw_odh-ai-gateway-infra(d618f1d1-4446-4931-8c4d-5886122097d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" Oct 30 22:15:04.837891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.837875 5607 scope.go:122] "RemoveContainer" containerID="7ea4dd3ae44d0630da0a5da9072b107b8a1f86cb311c8af74997489ba121cd2c" Oct 30 22:15:04.838258 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:04.838235 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz_odh-ai-gateway-infra(fbc03183-d218-450b-9e75-8acc338a1d98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" Oct 30 22:15:04.839443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.839423 5607 generic.go:356] "Generic (PLEG): container finished" podID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerID="148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9" exitCode=60 Oct 30 22:15:04.839545 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.839481 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerDied","Data":"148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9"} Oct 30 22:15:04.839794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.839779 5607 scope.go:122] "RemoveContainer" containerID="148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9" Oct 30 22:15:04.840090 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:04.840069 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-b9fql_odh-ai-gateway-infra(819b5d6f-8cbf-46f5-86cf-6cb193337970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" Oct 30 22:15:04.845336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.845309 5607 scope.go:122] "RemoveContainer" containerID="7954a688194229da8a5c1d373d5511c1b5bc3f62a56f7f6894edaba4180d200a" Oct 30 22:15:04.856850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.856833 5607 scope.go:122] "RemoveContainer" containerID="cd7c33743c2152a3e2d9db1d3c06874af244b45557d787ca776f8ab658983407" Oct 30 22:15:04.868345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.868330 5607 scope.go:122] "RemoveContainer" containerID="568759dec20c4d056103e766e3bb15c36a8934d363e7bdbef3326e9e763d0c38" Oct 30 22:15:04.948205 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:04.948175 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" podUID="232b4e73-0623-42a6-a244-90443cb59552" containerName="payload-pre-processing" containerID="cri-o://0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86" gracePeriod=25 Oct 30 22:15:05.227688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.227665 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:15:05.302134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.302104 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l\" (UniqueName: \"kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l\") pod \"232b4e73-0623-42a6-a244-90443cb59552\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " Oct 30 22:15:05.302299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.302222 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume\") pod \"232b4e73-0623-42a6-a244-90443cb59552\" (UID: \"232b4e73-0623-42a6-a244-90443cb59552\") " Oct 30 22:15:05.302560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.302538 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume" pod "232b4e73-0623-42a6-a244-90443cb59552" (UID: "232b4e73-0623-42a6-a244-90443cb59552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:05.302852 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.302834 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/232b4e73-0623-42a6-a244-90443cb59552-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:05.304409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.304387 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l" pod "232b4e73-0623-42a6-a244-90443cb59552" (UID: "232b4e73-0623-42a6-a244-90443cb59552"). InnerVolumeSpecName "kube-api-access-tsl9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:05.404054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.403995 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsl9l\" (UniqueName: \"kubernetes.io/projected/232b4e73-0623-42a6-a244-90443cb59552-kube-api-access-tsl9l\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:05.717144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.717114 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:15:05.859062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.854478 5607 scope.go:122] "RemoveContainer" containerID="148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9" Oct 30 22:15:05.859062 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:05.854800 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-b9fql_odh-ai-gateway-infra(819b5d6f-8cbf-46f5-86cf-6cb193337970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" Oct 30 22:15:05.859062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.856203 5607 scope.go:122] "RemoveContainer" containerID="4a0147ca60c590f22ddc4953853202db1a57190062083224a1e98cc878e48ead" Oct 30 22:15:05.859062 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:05.856484 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d_odh-ai-gateway-infra(225f43ca-e55f-49fe-822c-baac58e23d4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" Oct 30 22:15:05.859062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.858038 5607 scope.go:122] "RemoveContainer" containerID="5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc" Oct 30 22:15:05.859062 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:05.858316 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q_odh-ai-gateway-infra(fdb8d3dc-d327-46ef-a13a-7a443823ed14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" Oct 30 22:15:05.860294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.860275 5607 scope.go:122] "RemoveContainer" containerID="a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657" Oct 30 22:15:05.860662 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:05.860643 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw_odh-ai-gateway-infra(d618f1d1-4446-4931-8c4d-5886122097d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" Oct 30 22:15:05.861491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.861471 5607 generic.go:356] "Generic (PLEG): container finished" podID="232b4e73-0623-42a6-a244-90443cb59552" containerID="0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86" exitCode=0 Oct 30 22:15:05.861578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.861568 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" event={"ID":"232b4e73-0623-42a6-a244-90443cb59552","Type":"ContainerDied","Data":"0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86"} Oct 30 22:15:05.861635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.861601 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" event={"ID":"232b4e73-0623-42a6-a244-90443cb59552","Type":"ContainerDied","Data":"3a6b9347c940aa6dd62d77955de5cc35af57a4615d54c6e79453ad864d17b15b"} Oct 30 22:15:05.861635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.861620 5607 scope.go:122] "RemoveContainer" containerID="0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86" Oct 30 22:15:05.861727 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.861716 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf" Oct 30 22:15:05.874823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.874800 5607 scope.go:122] "RemoveContainer" containerID="0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86" Oct 30 22:15:05.875211 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:05.875189 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86\": container with ID starting with 0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86 not found: ID does not exist" containerID="0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86" Oct 30 22:15:05.875294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:05.875216 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86"} err="failed to get container status \"0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86\": rpc error: code = NotFound desc = could not find container \"0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86\": container with ID starting with 0917bc411f3ffcc8082009bdf8a58b12c75fd75e120f233a8f95bb71f1214f86 not found: ID does not exist" Oct 30 22:15:06.089868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:06.089786 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf"] Oct 30 22:15:06.096649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:06.096626 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04d404-5f8467dd95-cmncf"] Oct 30 22:15:06.871161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:06.871133 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232b4e73-0623-42a6-a244-90443cb59552" path="/var/lib/kubelet/pods/232b4e73-0623-42a6-a244-90443cb59552/volumes" Oct 30 22:15:07.731008 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:07.730984 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:15:09.798393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:09.798359 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b"] Oct 30 22:15:09.798840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:09.798586 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" podUID="06daf9c0-8241-440a-9770-e5040e3257d8" containerName="maas-api" containerID="cri-o://b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af" gracePeriod=30 Oct 30 22:15:10.023765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.023732 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d"] Oct 30 22:15:10.103309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.103285 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:15:10.157237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.157202 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs\") pod \"06daf9c0-8241-440a-9770-e5040e3257d8\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " Oct 30 22:15:10.157390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.157375 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx\" (UniqueName: \"kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx\") pod \"06daf9c0-8241-440a-9770-e5040e3257d8\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " Oct 30 22:15:10.157469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.157448 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\") pod \"06daf9c0-8241-440a-9770-e5040e3257d8\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " Oct 30 22:15:10.157604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.157588 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca\") pod \"06daf9c0-8241-440a-9770-e5040e3257d8\" (UID: \"06daf9c0-8241-440a-9770-e5040e3257d8\") " Oct 30 22:15:10.158081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.158046 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca" pod "06daf9c0-8241-440a-9770-e5040e3257d8" (UID: "06daf9c0-8241-440a-9770-e5040e3257d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:10.158366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.158325 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06daf9c0-8241-440a-9770-e5040e3257d8-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.160061 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.160035 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx" pod "06daf9c0-8241-440a-9770-e5040e3257d8" (UID: "06daf9c0-8241-440a-9770-e5040e3257d8"). InnerVolumeSpecName "kube-api-access-8rffx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:10.160891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.160857 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs" pod "06daf9c0-8241-440a-9770-e5040e3257d8" (UID: "06daf9c0-8241-440a-9770-e5040e3257d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:10.160990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.160889 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls" pod "06daf9c0-8241-440a-9770-e5040e3257d8" (UID: "06daf9c0-8241-440a-9770-e5040e3257d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:10.194895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.194871 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:10.259538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.259495 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr\" (UniqueName: \"kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr\") pod \"225f43ca-e55f-49fe-822c-baac58e23d4c\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " Oct 30 22:15:10.259730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.259645 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca\") pod \"225f43ca-e55f-49fe-822c-baac58e23d4c\" (UID: \"225f43ca-e55f-49fe-822c-baac58e23d4c\") " Oct 30 22:15:10.260071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.260042 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca" pod "225f43ca-e55f-49fe-822c-baac58e23d4c" (UID: "225f43ca-e55f-49fe-822c-baac58e23d4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:10.260220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.260197 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.260220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.260215 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rffx\" (UniqueName: \"kubernetes.io/projected/06daf9c0-8241-440a-9770-e5040e3257d8-kube-api-access-8rffx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.260307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.260230 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06daf9c0-8241-440a-9770-e5040e3257d8-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.260307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.260238 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/225f43ca-e55f-49fe-822c-baac58e23d4c-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.262710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.262685 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr" pod "225f43ca-e55f-49fe-822c-baac58e23d4c" (UID: "225f43ca-e55f-49fe-822c-baac58e23d4c"). InnerVolumeSpecName "kube-api-access-kpmmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:10.361695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.361618 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmmr\" (UniqueName: \"kubernetes.io/projected/225f43ca-e55f-49fe-822c-baac58e23d4c-kube-api-access-kpmmr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.899232 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.899203 5607 generic.go:356] "Generic (PLEG): container finished" podID="06daf9c0-8241-440a-9770-e5040e3257d8" containerID="b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af" exitCode=0 Oct 30 22:15:10.899646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.899245 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" Oct 30 22:15:10.899646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.899280 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" event={"ID":"06daf9c0-8241-440a-9770-e5040e3257d8","Type":"ContainerDied","Data":"b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af"} Oct 30 22:15:10.899646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.899330 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b" event={"ID":"06daf9c0-8241-440a-9770-e5040e3257d8","Type":"ContainerDied","Data":"16c03c43f8a92575e5430e3da43bddaaa14c9f4ba9dd6cb676073d83777f6b57"} Oct 30 22:15:10.899646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.899351 5607 scope.go:122] "RemoveContainer" containerID="b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af" Oct 30 22:15:10.901009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.900985 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" event={"ID":"225f43ca-e55f-49fe-822c-baac58e23d4c","Type":"ContainerDied","Data":"09bd192916d18af80672c9c8d4474293e44b8ad311ac0377705bce6c86b273aa"} Oct 30 22:15:10.901127 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.901077 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d" Oct 30 22:15:10.910782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.910758 5607 scope.go:122] "RemoveContainer" containerID="b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af" Oct 30 22:15:10.911072 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:10.911052 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af\": container with ID starting with b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af not found: ID does not exist" containerID="b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af" Oct 30 22:15:10.911147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.911086 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af"} err="failed to get container status \"b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af\": rpc error: code = NotFound desc = could not find container \"b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af\": container with ID starting with b8f31bb20a052710028f6e7f16881a852a2ae90962e7f206ecff4c23057738af not found: ID does not exist" Oct 30 22:15:10.911147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:10.911104 5607 scope.go:122] "RemoveContainer" containerID="4a0147ca60c590f22ddc4953853202db1a57190062083224a1e98cc878e48ead" Oct 30 22:15:11.076504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:11.076444 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b"] Oct 30 22:15:11.079172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:11.079150 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-872730-84d975656b-fwm8b"] Oct 30 22:15:11.191596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:11.191570 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d"] Oct 30 22:15:11.194148 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:11.194127 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-872730-29889975-xvb8d"] Oct 30 22:15:12.871125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:12.871093 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06daf9c0-8241-440a-9770-e5040e3257d8" path="/var/lib/kubelet/pods/06daf9c0-8241-440a-9770-e5040e3257d8/volumes" Oct 30 22:15:12.871522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:12.871473 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" path="/var/lib/kubelet/pods/225f43ca-e55f-49fe-822c-baac58e23d4c/volumes" Oct 30 22:15:15.865404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:15.865372 5607 scope.go:122] "RemoveContainer" containerID="5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc" Oct 30 22:15:16.956256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:16.956227 5607 generic.go:356] "Generic (PLEG): container finished" podID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerID="d7664e1f3b4739a44e2dc8dd88a7ce5e70b3761bf7235d3f2c64bd2f5dcdd7cf" exitCode=60 Oct 30 22:15:16.956639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:16.956294 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" event={"ID":"fdb8d3dc-d327-46ef-a13a-7a443823ed14","Type":"ContainerDied","Data":"d7664e1f3b4739a44e2dc8dd88a7ce5e70b3761bf7235d3f2c64bd2f5dcdd7cf"} Oct 30 22:15:16.956639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:16.956348 5607 scope.go:122] "RemoveContainer" containerID="5ce062da5692d9af2957b755d554d78359b3125597e83dc2669a838aa4137abc" Oct 30 22:15:16.956756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:16.956670 5607 scope.go:122] "RemoveContainer" containerID="d7664e1f3b4739a44e2dc8dd88a7ce5e70b3761bf7235d3f2c64bd2f5dcdd7cf" Oct 30 22:15:16.956902 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:16.956879 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q_odh-ai-gateway-infra(fdb8d3dc-d327-46ef-a13a-7a443823ed14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" Oct 30 22:15:17.862650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:17.862622 5607 scope.go:122] "RemoveContainer" containerID="148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9" Oct 30 22:15:17.862936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:17.862920 5607 scope.go:122] "RemoveContainer" containerID="a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657" Oct 30 22:15:18.091370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.091343 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q"] Oct 30 22:15:18.271841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.271813 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:18.366877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.366855 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca\") pod \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " Oct 30 22:15:18.367023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.366926 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh\" (UniqueName: \"kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh\") pod \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\" (UID: \"fdb8d3dc-d327-46ef-a13a-7a443823ed14\") " Oct 30 22:15:18.367260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.367238 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca" pod "fdb8d3dc-d327-46ef-a13a-7a443823ed14" (UID: "fdb8d3dc-d327-46ef-a13a-7a443823ed14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.367548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.367532 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb8d3dc-d327-46ef-a13a-7a443823ed14-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.368932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.368911 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh" pod "fdb8d3dc-d327-46ef-a13a-7a443823ed14" (UID: "fdb8d3dc-d327-46ef-a13a-7a443823ed14"). InnerVolumeSpecName "kube-api-access-w57qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.469208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.469183 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57qh\" (UniqueName: \"kubernetes.io/projected/fdb8d3dc-d327-46ef-a13a-7a443823ed14-kube-api-access-w57qh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.863529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.863506 5607 scope.go:122] "RemoveContainer" containerID="7ea4dd3ae44d0630da0a5da9072b107b8a1f86cb311c8af74997489ba121cd2c" Oct 30 22:15:18.984510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.984480 5607 generic.go:356] "Generic (PLEG): container finished" podID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerID="693f0261813b3a13358bcc6dc250cd0084c363f2f79e432d39841a8e09405796" exitCode=60 Oct 30 22:15:18.984642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.984547 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerDied","Data":"693f0261813b3a13358bcc6dc250cd0084c363f2f79e432d39841a8e09405796"} Oct 30 22:15:18.984642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.984607 5607 scope.go:122] "RemoveContainer" containerID="148a6bb6c01e7f37e6cc065df35d21bc7309d313040a8571dfeb703842a654b9" Oct 30 22:15:18.984923 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.984905 5607 scope.go:122] "RemoveContainer" containerID="693f0261813b3a13358bcc6dc250cd0084c363f2f79e432d39841a8e09405796" Oct 30 22:15:18.985218 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:18.985191 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-b9fql_odh-ai-gateway-infra(819b5d6f-8cbf-46f5-86cf-6cb193337970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" Oct 30 22:15:18.986268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.986246 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" Oct 30 22:15:18.986369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.986247 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q" event={"ID":"fdb8d3dc-d327-46ef-a13a-7a443823ed14","Type":"ContainerDied","Data":"e95749012ce4b6b0c9ff3e42472311e104a13a2d7a1205a9ad4d4b6af19c11d8"} Oct 30 22:15:18.988429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.988407 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerStarted","Data":"fcdd7716df4d14d8e2de7f6acdf0e39c5632f895bd47f5952e3b5009cc926d1f"} Oct 30 22:15:18.998475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:18.998457 5607 scope.go:122] "RemoveContainer" containerID="d7664e1f3b4739a44e2dc8dd88a7ce5e70b3761bf7235d3f2c64bd2f5dcdd7cf" Oct 30 22:15:19.230398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:19.230372 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q"] Oct 30 22:15:19.239228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:19.239207 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04d404-29889975-v7n4q"] Oct 30 22:15:20.010431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.010386 5607 generic.go:356] "Generic (PLEG): container finished" podID="fbc03183-d218-450b-9e75-8acc338a1d98" containerID="2580e9e9738631c6121995cf43f262f7dc7f7f915a5e03e35e007fef2ea454fc" exitCode=60 Oct 30 22:15:20.010627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.010520 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" event={"ID":"fbc03183-d218-450b-9e75-8acc338a1d98","Type":"ContainerDied","Data":"2580e9e9738631c6121995cf43f262f7dc7f7f915a5e03e35e007fef2ea454fc"} Oct 30 22:15:20.010704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.010626 5607 scope.go:122] "RemoveContainer" containerID="7ea4dd3ae44d0630da0a5da9072b107b8a1f86cb311c8af74997489ba121cd2c" Oct 30 22:15:20.011623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.011601 5607 scope.go:122] "RemoveContainer" containerID="2580e9e9738631c6121995cf43f262f7dc7f7f915a5e03e35e007fef2ea454fc" Oct 30 22:15:20.012425 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:20.012393 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz_odh-ai-gateway-infra(fbc03183-d218-450b-9e75-8acc338a1d98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" Oct 30 22:15:20.018348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.017689 5607 generic.go:356] "Generic (PLEG): container finished" podID="d618f1d1-4446-4931-8c4d-5886122097d2" containerID="fcdd7716df4d14d8e2de7f6acdf0e39c5632f895bd47f5952e3b5009cc926d1f" exitCode=60 Oct 30 22:15:20.018348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.017790 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerDied","Data":"fcdd7716df4d14d8e2de7f6acdf0e39c5632f895bd47f5952e3b5009cc926d1f"} Oct 30 22:15:20.021585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.021562 5607 scope.go:122] "RemoveContainer" containerID="fcdd7716df4d14d8e2de7f6acdf0e39c5632f895bd47f5952e3b5009cc926d1f" Oct 30 22:15:20.022135 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:20.022110 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw_odh-ai-gateway-infra(d618f1d1-4446-4931-8c4d-5886122097d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" Oct 30 22:15:20.042341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.042314 5607 scope.go:122] "RemoveContainer" containerID="a91e42917f2cb30c45c874400a5c825654da846c0291afac9f433d5b6bd39657" Oct 30 22:15:20.129590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.129560 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql"] Oct 30 22:15:20.319401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.319381 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:20.379687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.379648 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw"] Oct 30 22:15:20.423976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.420849 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca\") pod \"819b5d6f-8cbf-46f5-86cf-6cb193337970\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " Oct 30 22:15:20.423976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.422124 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2\" (UniqueName: \"kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2\") pod \"819b5d6f-8cbf-46f5-86cf-6cb193337970\" (UID: \"819b5d6f-8cbf-46f5-86cf-6cb193337970\") " Oct 30 22:15:20.423976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.422604 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca" pod "819b5d6f-8cbf-46f5-86cf-6cb193337970" (UID: "819b5d6f-8cbf-46f5-86cf-6cb193337970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.423976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.422796 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/819b5d6f-8cbf-46f5-86cf-6cb193337970-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.463907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.463185 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2" pod "819b5d6f-8cbf-46f5-86cf-6cb193337970" (UID: "819b5d6f-8cbf-46f5-86cf-6cb193337970"). InnerVolumeSpecName "kube-api-access-lfcq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.524479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.524432 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfcq2\" (UniqueName: \"kubernetes.io/projected/819b5d6f-8cbf-46f5-86cf-6cb193337970-kube-api-access-lfcq2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.872288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:20.872259 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" path="/var/lib/kubelet/pods/fdb8d3dc-d327-46ef-a13a-7a443823ed14/volumes" Oct 30 22:15:21.030311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.030274 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" event={"ID":"819b5d6f-8cbf-46f5-86cf-6cb193337970","Type":"ContainerDied","Data":"87662f74498156b9c0e759dd9641cbdaa74de394bd29a3752c6d4ae3ee15771b"} Oct 30 22:15:21.030454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.030350 5607 scope.go:122] "RemoveContainer" containerID="693f0261813b3a13358bcc6dc250cd0084c363f2f79e432d39841a8e09405796" Oct 30 22:15:21.030454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.030293 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql" Oct 30 22:15:21.163060 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.163032 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz"] Oct 30 22:15:21.187716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.187689 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:21.272826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.272796 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql"] Oct 30 22:15:21.285909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.285884 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-b9fql"] Oct 30 22:15:21.318254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.318234 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:21.335131 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.335105 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d\" (UniqueName: \"kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d\") pod \"d618f1d1-4446-4931-8c4d-5886122097d2\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " Oct 30 22:15:21.335456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.335198 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca\") pod \"d618f1d1-4446-4931-8c4d-5886122097d2\" (UID: \"d618f1d1-4446-4931-8c4d-5886122097d2\") " Oct 30 22:15:21.335595 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.335562 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca" pod "d618f1d1-4446-4931-8c4d-5886122097d2" (UID: "d618f1d1-4446-4931-8c4d-5886122097d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.335843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.335823 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d618f1d1-4446-4931-8c4d-5886122097d2-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.337481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.337462 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d" pod "d618f1d1-4446-4931-8c4d-5886122097d2" (UID: "d618f1d1-4446-4931-8c4d-5886122097d2"). InnerVolumeSpecName "kube-api-access-6dw4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.436442 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.436418 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca\") pod \"fbc03183-d218-450b-9e75-8acc338a1d98\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " Oct 30 22:15:21.436545 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.436494 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g\" (UniqueName: \"kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g\") pod \"fbc03183-d218-450b-9e75-8acc338a1d98\" (UID: \"fbc03183-d218-450b-9e75-8acc338a1d98\") " Oct 30 22:15:21.436787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.436766 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca" pod "fbc03183-d218-450b-9e75-8acc338a1d98" (UID: "fbc03183-d218-450b-9e75-8acc338a1d98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.437055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.437038 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dw4d\" (UniqueName: \"kubernetes.io/projected/d618f1d1-4446-4931-8c4d-5886122097d2-kube-api-access-6dw4d\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.437117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.437060 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc03183-d218-450b-9e75-8acc338a1d98-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.438481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.438462 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g" pod "fbc03183-d218-450b-9e75-8acc338a1d98" (UID: "fbc03183-d218-450b-9e75-8acc338a1d98"). InnerVolumeSpecName "kube-api-access-l9d7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.539146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:21.539115 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9d7g\" (UniqueName: \"kubernetes.io/projected/fbc03183-d218-450b-9e75-8acc338a1d98-kube-api-access-l9d7g\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.038278 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.038246 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" Oct 30 22:15:22.038454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.038245 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz" event={"ID":"fbc03183-d218-450b-9e75-8acc338a1d98","Type":"ContainerDied","Data":"f4409e5cb06eb7181fb71563072f9d42044ba58e292ceef2073172209f890da6"} Oct 30 22:15:22.038454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.038385 5607 scope.go:122] "RemoveContainer" containerID="2580e9e9738631c6121995cf43f262f7dc7f7f915a5e03e35e007fef2ea454fc" Oct 30 22:15:22.041004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.040982 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" Oct 30 22:15:22.041114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.041004 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw" event={"ID":"d618f1d1-4446-4931-8c4d-5886122097d2","Type":"ContainerDied","Data":"ee4bab1f434ba81f2fcccad5518ebf9e80d5d02d5785c57747757f203dd5deeb"} Oct 30 22:15:22.050714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.050674 5607 scope.go:122] "RemoveContainer" containerID="fcdd7716df4d14d8e2de7f6acdf0e39c5632f895bd47f5952e3b5009cc926d1f" Oct 30 22:15:22.208020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.207995 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz"] Oct 30 22:15:22.212064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.212036 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-427d70-29889975-xv8fz"] Oct 30 22:15:22.324480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.324417 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw"] Oct 30 22:15:22.327165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.327147 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889975-r9ntw"] Oct 30 22:15:22.864119 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:22.864080 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:15:22.871543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.871517 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" path="/var/lib/kubelet/pods/819b5d6f-8cbf-46f5-86cf-6cb193337970/volumes" Oct 30 22:15:22.871947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.871931 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" path="/var/lib/kubelet/pods/d618f1d1-4446-4931-8c4d-5886122097d2/volumes" Oct 30 22:15:22.872349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:22.872335 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" path="/var/lib/kubelet/pods/fbc03183-d218-450b-9e75-8acc338a1d98/volumes" Oct 30 22:15:32.741714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:32.741674 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9"] Oct 30 22:15:32.852150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:32.852113 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8"] Oct 30 22:15:37.742309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:37.742270 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" podUID="258368e2-ecb9-4902-9924-87c599d7078b" containerName="payload-processing" containerID="cri-o://7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d" gracePeriod=25 Oct 30 22:15:37.852990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:37.852909 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" podUID="c718af35-daac-4738-9785-dc80e68982e3" containerName="payload-pre-processing" containerID="cri-o://31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339" gracePeriod=25 Oct 30 22:15:38.043602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.043562 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:15:38.129291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.129266 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:15:38.146484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.146452 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume\") pod \"258368e2-ecb9-4902-9924-87c599d7078b\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " Oct 30 22:15:38.146666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.146599 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2\" (UniqueName: \"kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2\") pod \"258368e2-ecb9-4902-9924-87c599d7078b\" (UID: \"258368e2-ecb9-4902-9924-87c599d7078b\") " Oct 30 22:15:38.146856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.146830 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume" pod "258368e2-ecb9-4902-9924-87c599d7078b" (UID: "258368e2-ecb9-4902-9924-87c599d7078b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:38.146914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.146906 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258368e2-ecb9-4902-9924-87c599d7078b-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.148882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.148853 5607 generic.go:356] "Generic (PLEG): container finished" podID="c718af35-daac-4738-9785-dc80e68982e3" containerID="31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339" exitCode=0 Oct 30 22:15:38.149032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.148910 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" Oct 30 22:15:38.149032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.148967 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2" pod "258368e2-ecb9-4902-9924-87c599d7078b" (UID: "258368e2-ecb9-4902-9924-87c599d7078b"). InnerVolumeSpecName "kube-api-access-9shp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:38.149032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.148985 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" event={"ID":"c718af35-daac-4738-9785-dc80e68982e3","Type":"ContainerDied","Data":"31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339"} Oct 30 22:15:38.149218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.149031 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8" event={"ID":"c718af35-daac-4738-9785-dc80e68982e3","Type":"ContainerDied","Data":"23fb08ae7725c644f81b9b106c37b45739184b35ddd36cb54ecea90507951850"} Oct 30 22:15:38.149218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.149066 5607 scope.go:122] "RemoveContainer" containerID="31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339" Oct 30 22:15:38.150457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.150434 5607 generic.go:356] "Generic (PLEG): container finished" podID="258368e2-ecb9-4902-9924-87c599d7078b" containerID="7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d" exitCode=0 Oct 30 22:15:38.150534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.150463 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" Oct 30 22:15:38.150570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.150548 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" event={"ID":"258368e2-ecb9-4902-9924-87c599d7078b","Type":"ContainerDied","Data":"7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d"} Oct 30 22:15:38.150609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.150588 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9" event={"ID":"258368e2-ecb9-4902-9924-87c599d7078b","Type":"ContainerDied","Data":"9379ddcc6bd98a18dd276f5cf814a20186ca9d8fa0adb4ad32b43226439f9c8a"} Oct 30 22:15:38.159393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.159378 5607 scope.go:122] "RemoveContainer" containerID="31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339" Oct 30 22:15:38.159763 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:38.159731 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339\": container with ID starting with 31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339 not found: ID does not exist" containerID="31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339" Oct 30 22:15:38.159874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.159768 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339"} err="failed to get container status \"31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339\": rpc error: code = NotFound desc = could not find container \"31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339\": container with ID starting with 31366e9256797106872b38705033f39e931e3ae133b8244be2c74817b2381339 not found: ID does not exist" Oct 30 22:15:38.159874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.159792 5607 scope.go:122] "RemoveContainer" containerID="7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d" Oct 30 22:15:38.171889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.171868 5607 scope.go:122] "RemoveContainer" containerID="7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d" Oct 30 22:15:38.172321 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:38.172296 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d\": container with ID starting with 7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d not found: ID does not exist" containerID="7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d" Oct 30 22:15:38.172410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.172327 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d"} err="failed to get container status \"7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d\": rpc error: code = NotFound desc = could not find container \"7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d\": container with ID starting with 7e66eddcd34bed4429aa403163e788da2aba10a49c5d8c32882c851f6eb4743d not found: ID does not exist" Oct 30 22:15:38.247566 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.247529 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume\") pod \"c718af35-daac-4738-9785-dc80e68982e3\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " Oct 30 22:15:38.247777 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.247695 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd\" (UniqueName: \"kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd\") pod \"c718af35-daac-4738-9785-dc80e68982e3\" (UID: \"c718af35-daac-4738-9785-dc80e68982e3\") " Oct 30 22:15:38.247897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.247876 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume" pod "c718af35-daac-4738-9785-dc80e68982e3" (UID: "c718af35-daac-4738-9785-dc80e68982e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:38.248207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.248189 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c718af35-daac-4738-9785-dc80e68982e3-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.248291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.248207 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shp2\" (UniqueName: \"kubernetes.io/projected/258368e2-ecb9-4902-9924-87c599d7078b-kube-api-access-9shp2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.249985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.249962 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd" pod "c718af35-daac-4738-9785-dc80e68982e3" (UID: "c718af35-daac-4738-9785-dc80e68982e3"). InnerVolumeSpecName "kube-api-access-9brrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:38.279852 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.279780 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9"] Oct 30 22:15:38.283889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.283864 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4401bb-9675b947-89hg9"] Oct 30 22:15:38.349333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.349301 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brrd\" (UniqueName: \"kubernetes.io/projected/c718af35-daac-4738-9785-dc80e68982e3-kube-api-access-9brrd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.573044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.572935 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8"] Oct 30 22:15:38.579828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.579797 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4401bb-5d65df5747-5spv8"] Oct 30 22:15:38.871188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.871110 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258368e2-ecb9-4902-9924-87c599d7078b" path="/var/lib/kubelet/pods/258368e2-ecb9-4902-9924-87c599d7078b/volumes" Oct 30 22:15:38.871562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:38.871525 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c718af35-daac-4738-9785-dc80e68982e3" path="/var/lib/kubelet/pods/c718af35-daac-4738-9785-dc80e68982e3/volumes" Oct 30 22:15:40.210766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.210691 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59"] Oct 30 22:15:40.211288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211275 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.211333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211290 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.211333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211309 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.211333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211314 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.211333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211326 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.211333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211332 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211338 5607 cpu_manager.go:414] "containerMap: removing container" podUID="258368e2-ecb9-4902-9924-87c599d7078b" containerName="payload-processing" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211343 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258368e2-ecb9-4902-9924-87c599d7078b" containerName="payload-processing" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211351 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211355 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211363 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c718af35-daac-4738-9785-dc80e68982e3" containerName="payload-pre-processing" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211367 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c718af35-daac-4738-9785-dc80e68982e3" containerName="payload-pre-processing" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211379 5607 cpu_manager.go:414] "containerMap: removing container" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211383 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211389 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211393 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211405 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211409 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211416 5607 cpu_manager.go:414] "containerMap: removing container" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211420 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211432 5607 cpu_manager.go:414] "containerMap: removing container" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211437 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211443 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211447 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211455 5607 cpu_manager.go:414] "containerMap: removing container" podUID="232b4e73-0623-42a6-a244-90443cb59552" containerName="payload-pre-processing" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211460 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232b4e73-0623-42a6-a244-90443cb59552" containerName="payload-pre-processing" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211468 5607 cpu_manager.go:414] "containerMap: removing container" podUID="06daf9c0-8241-440a-9770-e5040e3257d8" containerName="maas-api" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211473 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06daf9c0-8241-440a-9770-e5040e3257d8" containerName="maas-api" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211480 5607 cpu_manager.go:414] "containerMap: removing container" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211484 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211491 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211495 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.211489 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211502 5607 cpu_manager.go:414] "containerMap: removing container" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211506 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211585 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211594 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211599 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211607 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f43ca-e55f-49fe-822c-baac58e23d4c" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211612 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211618 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="232b4e73-0623-42a6-a244-90443cb59552" containerName="payload-pre-processing" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211623 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="06daf9c0-8241-440a-9770-e5040e3257d8" containerName="maas-api" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211629 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211634 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc03183-d218-450b-9e75-8acc338a1d98" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211638 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211643 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211649 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211655 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211661 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="258368e2-ecb9-4902-9924-87c599d7078b" containerName="payload-processing" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211666 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:40.212255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.211671 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c718af35-daac-4738-9785-dc80e68982e3" containerName="payload-pre-processing" Oct 30 22:15:40.216785 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.216770 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.322713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.322687 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ad7da0-serving-cert\"" Oct 30 22:15:40.325604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.325580 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59"] Oct 30 22:15:40.374607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.374578 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.374772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.374641 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48566\" (UniqueName: \"kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.374772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.374716 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.374772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.374753 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.476367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.476294 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48566\" (UniqueName: \"kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.476367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.476329 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.476367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.476358 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.476589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.476504 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.477036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.477017 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.479254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.479235 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.479362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.479324 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.538764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.538732 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48566\" (UniqueName: \"kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566\") pod \"maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:40.830842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:40.830764 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:41.203666 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:41.203625 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd88396_ecd7_4f9e_9c36_cd2f3ad5deb8.slice/crio-cb8b5f2f7a2998275d510c30ba9651809b5842921410bef78761b7e0b6b97ea2 WatchSource:0}: Error finding container cb8b5f2f7a2998275d510c30ba9651809b5842921410bef78761b7e0b6b97ea2: Status 404 returned error can't find the container with id cb8b5f2f7a2998275d510c30ba9651809b5842921410bef78761b7e0b6b97ea2 Oct 30 22:15:41.205142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:41.205120 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:15:41.207322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:41.207298 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59"] Oct 30 22:15:42.184889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:42.184672 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" event={"ID":"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8","Type":"ContainerStarted","Data":"cb8b5f2f7a2998275d510c30ba9651809b5842921410bef78761b7e0b6b97ea2"} Oct 30 22:15:43.193415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:43.193374 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" event={"ID":"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8","Type":"ContainerStarted","Data":"ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72"} Oct 30 22:15:43.193874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:43.193440 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:43.314524 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:43.314474 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" podStartSLOduration=2.394523887 podStartE2EDuration="3.314460294s" totalImagesPullingTime="919.936407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:43.313428861 +0000 UTC m=+1722.713984983" watchObservedRunningTime="2026-10-30 22:15:43.314460294 +0000 UTC m=+1722.715016456" Oct 30 22:15:46.620167 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.620138 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9"] Oct 30 22:15:46.620846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.620829 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:46.620891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.620848 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d618f1d1-4446-4931-8c4d-5886122097d2" containerName="cleanup" Oct 30 22:15:46.621037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.621025 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="819b5d6f-8cbf-46f5-86cf-6cb193337970" containerName="cleanup" Oct 30 22:15:46.621078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.621045 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb8d3dc-d327-46ef-a13a-7a443823ed14" containerName="cleanup" Oct 30 22:15:46.624632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.624615 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650381 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650434 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650486 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650523 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650636 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650608 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7m5h\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650650 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650666 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650784 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.650853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.650839 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.671325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.671303 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74"] Oct 30 22:15:46.677382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.677358 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.687262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.687240 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4cf21208-openshift-default-dockercfg-js79h\"" Oct 30 22:15:46.730304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.730276 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9"] Oct 30 22:15:46.752021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.751947 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752061 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752095 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752118 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752142 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752321 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752359 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752440 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752423 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7m5h\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752466 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752493 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752524 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752525 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752598 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752653 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752709 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752716 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752758 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.752893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752767 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xhw\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752826 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.752893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752876 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.753211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.752914 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.753211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.753046 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.753512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.753442 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.755254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.755222 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.755373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.755283 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.799137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.799103 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74"] Oct 30 22:15:46.815329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.815298 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7m5h\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.818411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.818385 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.854482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854450 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854497 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854523 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854553 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854646 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854736 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854758 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xhw\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854782 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.854893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.854810 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.855142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.855018 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.855370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.855338 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.855477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.855389 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.855551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.855474 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.855618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.855544 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.857306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.857271 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.858291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.858263 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.917856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.917794 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xhw\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.918038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.918020 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:46.938918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.938898 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:46.996812 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:46.996212 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:47.394176 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.394140 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9"] Oct 30 22:15:47.423687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.423632 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:47.423797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.423730 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:47.423797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.423774 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:47.436738 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:47.436710 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2baf35_6241_446f_9963_61b6702f46ef.slice/crio-fbc08075b1b112cd9f678df1b96635bacb81c9c6dcfd1cd3845efc225f65ce24 WatchSource:0}: Error finding container fbc08075b1b112cd9f678df1b96635bacb81c9c6dcfd1cd3845efc225f65ce24: Status 404 returned error can't find the container with id fbc08075b1b112cd9f678df1b96635bacb81c9c6dcfd1cd3845efc225f65ce24 Oct 30 22:15:47.439585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.439553 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:47.439661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.439629 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:47.439661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.439657 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:47.440188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:47.440101 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74"] Oct 30 22:15:48.248229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.248189 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" event={"ID":"2d2baf35-6241-446f-9963-61b6702f46ef","Type":"ContainerStarted","Data":"eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07"} Oct 30 22:15:48.248804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.248247 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" event={"ID":"2d2baf35-6241-446f-9963-61b6702f46ef","Type":"ContainerStarted","Data":"fbc08075b1b112cd9f678df1b96635bacb81c9c6dcfd1cd3845efc225f65ce24"} Oct 30 22:15:48.250191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.250161 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" event={"ID":"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad","Type":"ContainerStarted","Data":"0aaefa7ebd87be897a41007b90713ab363b02243b795419972ac1970c2b3d7a5"} Oct 30 22:15:48.250319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.250209 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" event={"ID":"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad","Type":"ContainerStarted","Data":"5f67b684e24d700b4b3c6b8025c5d5ed49c72b371b12f63ab018c0532b032cf9"} Oct 30 22:15:48.493647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.493595 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podStartSLOduration=2.493581119 podStartE2EDuration="2.493581119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:48.44246456 +0000 UTC m=+1727.843020684" watchObservedRunningTime="2026-10-30 22:15:48.493581119 +0000 UTC m=+1727.894137239" Oct 30 22:15:48.571512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.571385 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podStartSLOduration=2.5713693490000002 podStartE2EDuration="2.571369349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:48.567889513 +0000 UTC m=+1727.968445643" watchObservedRunningTime="2026-10-30 22:15:48.571369349 +0000 UTC m=+1727.971925471" Oct 30 22:15:48.952034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.948315 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:48.952034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.950549 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:15:48.952034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.950611 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 22:15:48.998979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.998816 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:49.011985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:48.998939 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:15:49.011985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.003058 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 22:15:49.254340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.249822 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:15:49.780970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.780915 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5"] Oct 30 22:15:49.791407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.788263 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:49.854996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.850549 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5"] Oct 30 22:15:49.944037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.940919 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04d404-dockercfg-gnjfh\"" Oct 30 22:15:49.944037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.941289 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04d404\"" Oct 30 22:15:49.955030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.953416 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkl48\" (UniqueName: \"kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48\") pod \"payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:49.955030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.953538 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:49.955299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.955115 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:15:49.955299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:49.955191 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 22:15:50.029980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.028727 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:15:50.029980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.028783 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 22:15:50.054330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.054254 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkl48\" (UniqueName: \"kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48\") pod \"payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:50.054485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.054354 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:50.055164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.055144 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:50.153977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.151862 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkl48\" (UniqueName: \"kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48\") pod \"payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:50.155821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.155794 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:50.587318 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:15:50.586422 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5660253c_f8a7_408e_8d85_b85e624eb4b9.slice/crio-b495e9e056f408c594407e928cac2939734d5891cbeab2bc675d5c0565872872 WatchSource:0}: Error finding container b495e9e056f408c594407e928cac2939734d5891cbeab2bc675d5c0565872872: Status 404 returned error can't find the container with id b495e9e056f408c594407e928cac2939734d5891cbeab2bc675d5c0565872872 Oct 30 22:15:50.587766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.587515 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5"] Oct 30 22:15:50.948097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.939300 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:15:50.948097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:50.939361 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 22:15:51.002979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:51.000791 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:15:51.002979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:51.000850 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 22:15:51.323034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:51.313582 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" event={"ID":"5660253c-f8a7-408e-8d85-b85e624eb4b9","Type":"ContainerStarted","Data":"b495e9e056f408c594407e928cac2939734d5891cbeab2bc675d5c0565872872"} Oct 30 22:15:51.949385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:51.949344 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:15:51.949938 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:51.949409 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 22:15:52.007022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:52.006876 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:15:52.007022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:52.006932 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 22:15:52.881210 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:15:52.881174 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:15:52.943643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:52.943197 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:15:52.943643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:52.943253 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 22:15:53.018977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:53.018910 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:15:53.019494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:53.018995 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 22:15:54.199975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.199556 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:54.200424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.200300 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:54.201717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.201665 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:15:54.247913 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.247814 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:54.248354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.248333 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:54.249574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.249549 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:15:54.372891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.372854 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" event={"ID":"5660253c-f8a7-408e-8d85-b85e624eb4b9","Type":"ContainerStarted","Data":"254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25"} Oct 30 22:15:54.373195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.373178 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:15:54.692248 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:15:54.692191 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" podStartSLOduration=2.2923192070000002 podStartE2EDuration="5.69217751s" totalImagesPullingTime="3.399858303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:54.688923685 +0000 UTC m=+1734.089479811" watchObservedRunningTime="2026-10-30 22:15:54.69217751 +0000 UTC m=+1734.092733630" Oct 30 22:16:01.777785 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.777746 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h"] Oct 30 22:16:01.805816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.802245 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876055 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876098 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876199 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876227 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876321 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876409 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876509 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876569 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5xk\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.876671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.876638 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.902815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.902783 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h"] Oct 30 22:16:01.977546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977474 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.977733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977560 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.977733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977584 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.977733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977619 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5xk\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.977733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977669 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.977733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977701 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977736 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977791 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.977829 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.978299 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.978448 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.978436 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.978580 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.978613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.978461 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.980372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.980345 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:01.980454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:01.980403 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:02.040621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.040597 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:02.040711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.040628 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5xk\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:02.134520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.134492 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:02.557645 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:16:02.557608 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ee2745_6993_4e15_911b_bcee91b54dc5.slice/crio-dded8a5b2c0e59b819383ca12a686888c314f01d1885775f794938381e5781c7 WatchSource:0}: Error finding container dded8a5b2c0e59b819383ca12a686888c314f01d1885775f794938381e5781c7: Status 404 returned error can't find the container with id dded8a5b2c0e59b819383ca12a686888c314f01d1885775f794938381e5781c7 Oct 30 22:16:02.560614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.560578 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:02.560702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.560646 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:02.560702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.560675 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:02.561294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:02.561272 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h"] Oct 30 22:16:03.438291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:03.438250 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" event={"ID":"60ee2745-6993-4e15-911b-bcee91b54dc5","Type":"ContainerStarted","Data":"e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62"} Oct 30 22:16:03.438706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:03.438307 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" event={"ID":"60ee2745-6993-4e15-911b-bcee91b54dc5","Type":"ContainerStarted","Data":"dded8a5b2c0e59b819383ca12a686888c314f01d1885775f794938381e5781c7"} Oct 30 22:16:03.625692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:03.625628 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podStartSLOduration=2.625612194 podStartE2EDuration="2.625612194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.617517734 +0000 UTC m=+1743.018073858" watchObservedRunningTime="2026-10-30 22:16:03.625612194 +0000 UTC m=+1743.026168381" Oct 30 22:16:04.135501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:04.135465 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:04.137430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:04.137400 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:16:04.137622 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:04.137455 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:16:05.135704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:05.135669 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:16:05.136198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:05.135740 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:16:05.382599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:05.382572 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:16:06.134853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:06.134822 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.135074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:06.134887 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:16:07.135474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:07.135437 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.135882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:07.135522 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:16:08.135526 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:08.135490 5607 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:16:08.136056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:08.135551 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:16:09.223690 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:09.223658 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:09.225006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:09.224978 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:09.226165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:09.226141 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:16:17.666638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.666601 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm"] Oct 30 22:16:17.672276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.672256 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:17.748576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.748546 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm"] Oct 30 22:16:17.796495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.796454 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8b4b54\"" Oct 30 22:16:17.796934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.796915 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8b4b54-dockercfg-6slfl\"" Oct 30 22:16:17.834797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.834753 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw"] Oct 30 22:16:17.840374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.840346 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqhn\" (UniqueName: \"kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:17.840557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.840391 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:17.842131 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.842109 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:17.877478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.877446 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw"] Oct 30 22:16:17.947002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.946940 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqhn\" (UniqueName: \"kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:17.947203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.947027 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:17.947203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.947141 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2br\" (UniqueName: \"kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br\") pod \"payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:17.947408 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.947387 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:17.949709 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:17.949678 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:18.030231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.030204 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqhn\" (UniqueName: \"kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:18.049113 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.049072 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2br\" (UniqueName: \"kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br\") pod \"payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:18.049301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.049251 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:18.049897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.049873 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:18.112519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.112485 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2br\" (UniqueName: \"kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br\") pod \"payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:18.186669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.186645 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:18.289041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.287543 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:18.558444 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:16:18.558417 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d2487b_1050_4d75_9c7a_ed6ae2114e70.slice/crio-1a12a7b6dea479ff0e94e61d7d4bc625a849b69853a73be7926583ce85fd6cef WatchSource:0}: Error finding container 1a12a7b6dea479ff0e94e61d7d4bc625a849b69853a73be7926583ce85fd6cef: Status 404 returned error can't find the container with id 1a12a7b6dea479ff0e94e61d7d4bc625a849b69853a73be7926583ce85fd6cef Oct 30 22:16:18.561311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.561291 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw"] Oct 30 22:16:18.602305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.602268 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" event={"ID":"a5d2487b-1050-4d75-9c7a-ed6ae2114e70","Type":"ContainerStarted","Data":"1a12a7b6dea479ff0e94e61d7d4bc625a849b69853a73be7926583ce85fd6cef"} Oct 30 22:16:18.660785 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:16:18.660761 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8314d6c5_c110_4855_8ca6_f101d7b8a9f3.slice/crio-9a956bb27bb40da0ddcc354765cc1137568e6bf902d2331825ccf66e913c2cdb WatchSource:0}: Error finding container 9a956bb27bb40da0ddcc354765cc1137568e6bf902d2331825ccf66e913c2cdb: Status 404 returned error can't find the container with id 9a956bb27bb40da0ddcc354765cc1137568e6bf902d2331825ccf66e913c2cdb Oct 30 22:16:18.665122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:18.665100 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm"] Oct 30 22:16:19.610362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.610319 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" event={"ID":"a5d2487b-1050-4d75-9c7a-ed6ae2114e70","Type":"ContainerStarted","Data":"b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b"} Oct 30 22:16:19.610808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.610429 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:19.612029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.612004 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" event={"ID":"8314d6c5-c110-4855-8ca6-f101d7b8a9f3","Type":"ContainerStarted","Data":"40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3"} Oct 30 22:16:19.612175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.612045 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" event={"ID":"8314d6c5-c110-4855-8ca6-f101d7b8a9f3","Type":"ContainerStarted","Data":"9a956bb27bb40da0ddcc354765cc1137568e6bf902d2331825ccf66e913c2cdb"} Oct 30 22:16:19.612175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.612156 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:19.729325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.729253 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" podStartSLOduration=2.729236032 podStartE2EDuration="2.729236032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:19.728003718 +0000 UTC m=+1759.128559840" watchObservedRunningTime="2026-10-30 22:16:19.729236032 +0000 UTC m=+1759.129792153" Oct 30 22:16:19.848328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:19.847692 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" podStartSLOduration=2.847672782 podStartE2EDuration="2.847672782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:19.846286017 +0000 UTC m=+1759.246842140" watchObservedRunningTime="2026-10-30 22:16:19.847672782 +0000 UTC m=+1759.248228903" Oct 30 22:16:22.867729 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:22.867685 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:16:26.140386 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:26.140344 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655"] Oct 30 22:16:26.256032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:26.256001 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh"] Oct 30 22:16:28.859361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:28.859326 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf"] Oct 30 22:16:28.978306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:28.978273 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk"] Oct 30 22:16:30.635874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:30.635844 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:16:30.639133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:30.639102 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:16:31.141673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.141631 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" podUID="ea75a11f-47ab-414b-b350-9e870be08c99" containerName="payload-processing" containerID="cri-o://4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2" gracePeriod=25 Oct 30 22:16:31.257127 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.257057 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" podUID="ce3b65ac-f714-4bd1-9c7e-7feefc36316b" containerName="payload-pre-processing" containerID="cri-o://a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c" gracePeriod=25 Oct 30 22:16:31.425547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.425520 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:16:31.542056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.542033 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume\") pod \"ea75a11f-47ab-414b-b350-9e870be08c99\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " Oct 30 22:16:31.542212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.542095 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82\" (UniqueName: \"kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82\") pod \"ea75a11f-47ab-414b-b350-9e870be08c99\" (UID: \"ea75a11f-47ab-414b-b350-9e870be08c99\") " Oct 30 22:16:31.542475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.542446 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume" pod "ea75a11f-47ab-414b-b350-9e870be08c99" (UID: "ea75a11f-47ab-414b-b350-9e870be08c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:31.542769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.542748 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75a11f-47ab-414b-b350-9e870be08c99-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.544565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.544538 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82" pod "ea75a11f-47ab-414b-b350-9e870be08c99" (UID: "ea75a11f-47ab-414b-b350-9e870be08c99"). InnerVolumeSpecName "kube-api-access-hpx82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:31.565026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.564992 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:16:31.644106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.644014 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn\" (UniqueName: \"kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn\") pod \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " Oct 30 22:16:31.644581 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.644180 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume\") pod \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\" (UID: \"ce3b65ac-f714-4bd1-9c7e-7feefc36316b\") " Oct 30 22:16:31.644581 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.644536 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume" pod "ce3b65ac-f714-4bd1-9c7e-7feefc36316b" (UID: "ce3b65ac-f714-4bd1-9c7e-7feefc36316b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:31.644760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.644740 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.644817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.644765 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpx82\" (UniqueName: \"kubernetes.io/projected/ea75a11f-47ab-414b-b350-9e870be08c99-kube-api-access-hpx82\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.646341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.646319 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn" pod "ce3b65ac-f714-4bd1-9c7e-7feefc36316b" (UID: "ce3b65ac-f714-4bd1-9c7e-7feefc36316b"). InnerVolumeSpecName "kube-api-access-4jchn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:31.702293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.702259 5607 generic.go:356] "Generic (PLEG): container finished" podID="ce3b65ac-f714-4bd1-9c7e-7feefc36316b" containerID="a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c" exitCode=0 Oct 30 22:16:31.702500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.702310 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" Oct 30 22:16:31.702500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.702342 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" event={"ID":"ce3b65ac-f714-4bd1-9c7e-7feefc36316b","Type":"ContainerDied","Data":"a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c"} Oct 30 22:16:31.702500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.702467 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh" event={"ID":"ce3b65ac-f714-4bd1-9c7e-7feefc36316b","Type":"ContainerDied","Data":"11f9a2cc8672c3d977bb1e3241d85f2af154358dd1ba3847d68d71f7f9fabe28"} Oct 30 22:16:31.702500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.702499 5607 scope.go:122] "RemoveContainer" containerID="a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c" Oct 30 22:16:31.704220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.704198 5607 generic.go:356] "Generic (PLEG): container finished" podID="ea75a11f-47ab-414b-b350-9e870be08c99" containerID="4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2" exitCode=0 Oct 30 22:16:31.704331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.704263 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" event={"ID":"ea75a11f-47ab-414b-b350-9e870be08c99","Type":"ContainerDied","Data":"4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2"} Oct 30 22:16:31.704331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.704306 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" event={"ID":"ea75a11f-47ab-414b-b350-9e870be08c99","Type":"ContainerDied","Data":"17ebc60301105380e7279f6ea1076334e8a45a3f23ac409c571aa16c9986f383"} Oct 30 22:16:31.704540 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.704524 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655" Oct 30 22:16:31.713439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.713372 5607 scope.go:122] "RemoveContainer" containerID="a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c" Oct 30 22:16:31.713631 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:31.713605 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c\": container with ID starting with a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c not found: ID does not exist" containerID="a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c" Oct 30 22:16:31.713694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.713643 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c"} err="failed to get container status \"a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c\": rpc error: code = NotFound desc = could not find container \"a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c\": container with ID starting with a7718ccb33f7110f2e2c1a6f9a75b0aab8bf41dbb5e75aee550f4e5d96e5d42c not found: ID does not exist" Oct 30 22:16:31.713694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.713668 5607 scope.go:122] "RemoveContainer" containerID="4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2" Oct 30 22:16:31.726983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.726945 5607 scope.go:122] "RemoveContainer" containerID="4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2" Oct 30 22:16:31.727281 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:31.727258 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2\": container with ID starting with 4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2 not found: ID does not exist" containerID="4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2" Oct 30 22:16:31.727381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.727287 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2"} err="failed to get container status \"4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2\": rpc error: code = NotFound desc = could not find container \"4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2\": container with ID starting with 4ea860cdca1a550f1cc57a9e7d8f6b0807dbe7398f09a80dc04073e320937dd2 not found: ID does not exist" Oct 30 22:16:31.746250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.746221 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jchn\" (UniqueName: \"kubernetes.io/projected/ce3b65ac-f714-4bd1-9c7e-7feefc36316b-kube-api-access-4jchn\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.871929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.871896 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655"] Oct 30 22:16:31.874733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.874713 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cf21208-867cd99b5f-dp655"] Oct 30 22:16:31.985499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.985461 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh"] Oct 30 22:16:31.989114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:31.989092 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cf21208-8455c87ff9-627fh"] Oct 30 22:16:32.871606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:32.871569 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3b65ac-f714-4bd1-9c7e-7feefc36316b" path="/var/lib/kubelet/pods/ce3b65ac-f714-4bd1-9c7e-7feefc36316b/volumes" Oct 30 22:16:32.872009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:32.871999 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea75a11f-47ab-414b-b350-9e870be08c99" path="/var/lib/kubelet/pods/ea75a11f-47ab-414b-b350-9e870be08c99/volumes" Oct 30 22:16:33.860632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:33.860589 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" podUID="a60bc0c4-0d00-40c5-a610-0e247043568d" containerName="payload-processing" containerID="cri-o://b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d" gracePeriod=25 Oct 30 22:16:33.978722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:33.978682 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" podUID="38fc515f-55f6-49ac-815d-6b20ab486861" containerName="payload-pre-processing" containerID="cri-o://adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853" gracePeriod=25 Oct 30 22:16:34.149188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.149162 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:16:34.267981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.267788 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:16:34.283137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.283108 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv\" (UniqueName: \"kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv\") pod \"a60bc0c4-0d00-40c5-a610-0e247043568d\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " Oct 30 22:16:34.283337 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.283323 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume\") pod \"a60bc0c4-0d00-40c5-a610-0e247043568d\" (UID: \"a60bc0c4-0d00-40c5-a610-0e247043568d\") " Oct 30 22:16:34.283735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.283711 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume" pod "a60bc0c4-0d00-40c5-a610-0e247043568d" (UID: "a60bc0c4-0d00-40c5-a610-0e247043568d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:34.283867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.283849 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60bc0c4-0d00-40c5-a610-0e247043568d-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.286010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.285984 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv" pod "a60bc0c4-0d00-40c5-a610-0e247043568d" (UID: "a60bc0c4-0d00-40c5-a610-0e247043568d"). InnerVolumeSpecName "kube-api-access-qt6zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.385067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.384991 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume\") pod \"38fc515f-55f6-49ac-815d-6b20ab486861\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " Oct 30 22:16:34.385216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.385099 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs\" (UniqueName: \"kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs\") pod \"38fc515f-55f6-49ac-815d-6b20ab486861\" (UID: \"38fc515f-55f6-49ac-815d-6b20ab486861\") " Oct 30 22:16:34.385397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.385374 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume" pod "38fc515f-55f6-49ac-815d-6b20ab486861" (UID: "38fc515f-55f6-49ac-815d-6b20ab486861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:34.385458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.385398 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt6zv\" (UniqueName: \"kubernetes.io/projected/a60bc0c4-0d00-40c5-a610-0e247043568d-kube-api-access-qt6zv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.387302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.387278 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs" pod "38fc515f-55f6-49ac-815d-6b20ab486861" (UID: "38fc515f-55f6-49ac-815d-6b20ab486861"). InnerVolumeSpecName "kube-api-access-qlpbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:34.486987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.486938 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38fc515f-55f6-49ac-815d-6b20ab486861-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.486987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.486981 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpbs\" (UniqueName: \"kubernetes.io/projected/38fc515f-55f6-49ac-815d-6b20ab486861-kube-api-access-qlpbs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:34.726662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.726628 5607 generic.go:356] "Generic (PLEG): container finished" podID="38fc515f-55f6-49ac-815d-6b20ab486861" containerID="adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853" exitCode=0 Oct 30 22:16:34.726818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.726679 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" Oct 30 22:16:34.726818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.726714 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" event={"ID":"38fc515f-55f6-49ac-815d-6b20ab486861","Type":"ContainerDied","Data":"adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853"} Oct 30 22:16:34.726818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.726770 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk" event={"ID":"38fc515f-55f6-49ac-815d-6b20ab486861","Type":"ContainerDied","Data":"7bed2bd8e6c5f6b97a2dd0474fbae3407b52a12c4cb1084baf57a405d011e10e"} Oct 30 22:16:34.726818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.726792 5607 scope.go:122] "RemoveContainer" containerID="adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853" Oct 30 22:16:34.728417 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.728391 5607 generic.go:356] "Generic (PLEG): container finished" podID="a60bc0c4-0d00-40c5-a610-0e247043568d" containerID="b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d" exitCode=0 Oct 30 22:16:34.728565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.728449 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" Oct 30 22:16:34.728565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.728480 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" event={"ID":"a60bc0c4-0d00-40c5-a610-0e247043568d","Type":"ContainerDied","Data":"b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d"} Oct 30 22:16:34.728565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.728542 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf" event={"ID":"a60bc0c4-0d00-40c5-a610-0e247043568d","Type":"ContainerDied","Data":"60ab3e7ef4d6dce84f15a2d5af1239a634321af866e1fc290a70c378a3c41a7a"} Oct 30 22:16:34.749210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.749190 5607 scope.go:122] "RemoveContainer" containerID="adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853" Oct 30 22:16:34.749504 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:34.749484 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853\": container with ID starting with adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853 not found: ID does not exist" containerID="adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853" Oct 30 22:16:34.749578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.749511 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853"} err="failed to get container status \"adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853\": rpc error: code = NotFound desc = could not find container \"adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853\": container with ID starting with adae66116b78a88a2a852c97df1f831d33bb3dbbeff7d0bb042f6cc02cdd7853 not found: ID does not exist" Oct 30 22:16:34.749578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.749529 5607 scope.go:122] "RemoveContainer" containerID="b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d" Oct 30 22:16:34.758652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.758616 5607 scope.go:122] "RemoveContainer" containerID="b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d" Oct 30 22:16:34.758918 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:34.758893 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d\": container with ID starting with b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d not found: ID does not exist" containerID="b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d" Oct 30 22:16:34.758996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.758922 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d"} err="failed to get container status \"b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d\": rpc error: code = NotFound desc = could not find container \"b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d\": container with ID starting with b19747dfa5a5e03e6cedc32e67e5ea2de11ad109bd3c39861c8070749127ae5d not found: ID does not exist" Oct 30 22:16:34.895966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.895925 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk"] Oct 30 22:16:34.947049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:34.947025 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-427d70-544c944bc-b7qnk"] Oct 30 22:16:35.011726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:35.011666 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf"] Oct 30 22:16:35.016251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:35.016230 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-427d70-5f6f8598f-td4cf"] Oct 30 22:16:36.870555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:36.870527 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fc515f-55f6-49ac-815d-6b20ab486861" path="/var/lib/kubelet/pods/38fc515f-55f6-49ac-815d-6b20ab486861/volumes" Oct 30 22:16:36.870936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:36.870910 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60bc0c4-0d00-40c5-a610-0e247043568d" path="/var/lib/kubelet/pods/a60bc0c4-0d00-40c5-a610-0e247043568d/volumes" Oct 30 22:16:37.698747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:37.698711 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd"] Oct 30 22:16:37.699024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:37.698995 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" containerID="cri-o://01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400" gracePeriod=30 Oct 30 22:16:38.722984 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:38.722938 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q"] Oct 30 22:16:38.834675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:38.834644 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648"] Oct 30 22:16:42.982375 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:42.982317 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:16:43.010378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010352 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010471 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010569 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010620 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010625 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:43.010752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010690 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010742 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010775 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.010857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010823 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:43.010857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010825 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.011070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010933 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token\") pod \"212ef815-88f1-42d0-820d-d20ff273e1a6\" (UID: \"212ef815-88f1-42d0-820d-d20ff273e1a6\") " Oct 30 22:16:43.011070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.010940 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:43.011070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011057 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:43.011234 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011079 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:43.011512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011459 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.011512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011479 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ef815-88f1-42d0-820d-d20ff273e1a6-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.011512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011488 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.011512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011496 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.011512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.011503 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.012850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.012818 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:43.013037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.012986 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "kube-api-access-6wpm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:43.013156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.013135 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:43.013311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.013291 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token" pod "212ef815-88f1-42d0-820d-d20ff273e1a6" (UID: "212ef815-88f1-42d0-820d-d20ff273e1a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:43.112580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.112544 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.112580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.112579 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wpm6\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-kube-api-access-6wpm6\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.112580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.112590 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.112749 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.112599 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ef815-88f1-42d0-820d-d20ff273e1a6-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.723682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.723627 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" podUID="2f090d42-ab12-4e84-af58-5370016c2733" containerName="payload-processing" containerID="cri-o://39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba" gracePeriod=25 Oct 30 22:16:43.794620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.794588 5607 generic.go:356] "Generic (PLEG): container finished" podID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerID="01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400" exitCode=0 Oct 30 22:16:43.794736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.794642 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" Oct 30 22:16:43.794736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.794677 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" event={"ID":"212ef815-88f1-42d0-820d-d20ff273e1a6","Type":"ContainerDied","Data":"01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400"} Oct 30 22:16:43.794823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.794765 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd" event={"ID":"212ef815-88f1-42d0-820d-d20ff273e1a6","Type":"ContainerDied","Data":"a83030af7eda54e2f4733be301860a125809a72edf368b337bff430d4de56632"} Oct 30 22:16:43.794823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.794789 5607 scope.go:122] "RemoveContainer" containerID="01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400" Oct 30 22:16:43.835028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.834967 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" podUID="f25cea18-75f3-443d-957d-7d32ff2a1322" containerName="payload-pre-processing" containerID="cri-o://db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113" gracePeriod=25 Oct 30 22:16:43.835749 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.835733 5607 scope.go:122] "RemoveContainer" containerID="01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400" Oct 30 22:16:43.836067 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:43.836045 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400\": container with ID starting with 01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400 not found: ID does not exist" containerID="01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400" Oct 30 22:16:43.836129 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.836077 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400"} err="failed to get container status \"01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400\": rpc error: code = NotFound desc = could not find container \"01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400\": container with ID starting with 01735f123571515a5ae6328a504df4495981ce7abdb65ff6e8e901a07fd13400 not found: ID does not exist" Oct 30 22:16:43.969029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.968997 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd"] Oct 30 22:16:43.975499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:43.975439 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-9dpsd"] Oct 30 22:16:44.017842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.017815 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:16:44.124032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.124004 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume\") pod \"2f090d42-ab12-4e84-af58-5370016c2733\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " Oct 30 22:16:44.124137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.124108 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj\" (UniqueName: \"kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj\") pod \"2f090d42-ab12-4e84-af58-5370016c2733\" (UID: \"2f090d42-ab12-4e84-af58-5370016c2733\") " Oct 30 22:16:44.124431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.124408 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume" pod "2f090d42-ab12-4e84-af58-5370016c2733" (UID: "2f090d42-ab12-4e84-af58-5370016c2733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:44.124860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.124837 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f090d42-ab12-4e84-af58-5370016c2733-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:44.126442 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.126416 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj" pod "2f090d42-ab12-4e84-af58-5370016c2733" (UID: "2f090d42-ab12-4e84-af58-5370016c2733"). InnerVolumeSpecName "kube-api-access-lfvjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:44.128909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.128891 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:16:44.225246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.225215 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9\" (UniqueName: \"kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9\") pod \"f25cea18-75f3-443d-957d-7d32ff2a1322\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " Oct 30 22:16:44.225419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.225307 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume\") pod \"f25cea18-75f3-443d-957d-7d32ff2a1322\" (UID: \"f25cea18-75f3-443d-957d-7d32ff2a1322\") " Oct 30 22:16:44.225676 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.225627 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume" pod "f25cea18-75f3-443d-957d-7d32ff2a1322" (UID: "f25cea18-75f3-443d-957d-7d32ff2a1322"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:44.225875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.225861 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvjj\" (UniqueName: \"kubernetes.io/projected/2f090d42-ab12-4e84-af58-5370016c2733-kube-api-access-lfvjj\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:44.225927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.225877 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25cea18-75f3-443d-957d-7d32ff2a1322-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:44.227486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.227468 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9" pod "f25cea18-75f3-443d-957d-7d32ff2a1322" (UID: "f25cea18-75f3-443d-957d-7d32ff2a1322"). InnerVolumeSpecName "kube-api-access-zpnn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:44.327551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.327518 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnn9\" (UniqueName: \"kubernetes.io/projected/f25cea18-75f3-443d-957d-7d32ff2a1322-kube-api-access-zpnn9\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:44.803450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.803422 5607 generic.go:356] "Generic (PLEG): container finished" podID="f25cea18-75f3-443d-957d-7d32ff2a1322" containerID="db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113" exitCode=0 Oct 30 22:16:44.803653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.803467 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" Oct 30 22:16:44.803653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.803510 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" event={"ID":"f25cea18-75f3-443d-957d-7d32ff2a1322","Type":"ContainerDied","Data":"db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113"} Oct 30 22:16:44.803653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.803563 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648" event={"ID":"f25cea18-75f3-443d-957d-7d32ff2a1322","Type":"ContainerDied","Data":"18f0f041938f82b416eb7615ed20cb8f3b4afa3b9534b5e5647dfd0a2d20cdeb"} Oct 30 22:16:44.803653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.803590 5607 scope.go:122] "RemoveContainer" containerID="db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113" Oct 30 22:16:44.805208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.805183 5607 generic.go:356] "Generic (PLEG): container finished" podID="2f090d42-ab12-4e84-af58-5370016c2733" containerID="39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba" exitCode=0 Oct 30 22:16:44.805332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.805275 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" event={"ID":"2f090d42-ab12-4e84-af58-5370016c2733","Type":"ContainerDied","Data":"39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba"} Oct 30 22:16:44.805332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.805312 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" event={"ID":"2f090d42-ab12-4e84-af58-5370016c2733","Type":"ContainerDied","Data":"6f26dd7725840a80431a213041290fa154bffbec12b414b3c517c952dad48e54"} Oct 30 22:16:44.805332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.805320 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q" Oct 30 22:16:44.815549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.815534 5607 scope.go:122] "RemoveContainer" containerID="db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113" Oct 30 22:16:44.815804 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:44.815787 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113\": container with ID starting with db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113 not found: ID does not exist" containerID="db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113" Oct 30 22:16:44.815860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.815813 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113"} err="failed to get container status \"db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113\": rpc error: code = NotFound desc = could not find container \"db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113\": container with ID starting with db42908f70ac7852a6ce9b63f28e4cb44aff39f86cc6f230c2cd6c143e478113 not found: ID does not exist" Oct 30 22:16:44.815860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.815828 5607 scope.go:122] "RemoveContainer" containerID="39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba" Oct 30 22:16:44.824817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.824799 5607 scope.go:122] "RemoveContainer" containerID="39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba" Oct 30 22:16:44.825188 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:44.825167 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba\": container with ID starting with 39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba not found: ID does not exist" containerID="39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba" Oct 30 22:16:44.825258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.825195 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba"} err="failed to get container status \"39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba\": rpc error: code = NotFound desc = could not find container \"39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba\": container with ID starting with 39792c467cece3a4889212067264ef9da674a93e3018a56d4c48c0c67138adba not found: ID does not exist" Oct 30 22:16:44.870886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.870861 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" path="/var/lib/kubelet/pods/212ef815-88f1-42d0-820d-d20ff273e1a6/volumes" Oct 30 22:16:44.974261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.974238 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q"] Oct 30 22:16:44.976797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:44.976778 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-7455d77d87-tqp9q"] Oct 30 22:16:45.092036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:45.091981 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648"] Oct 30 22:16:45.096936 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:45.096916 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ad7da0-949bddf7f-b9648"] Oct 30 22:16:46.871170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:46.871127 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f090d42-ab12-4e84-af58-5370016c2733" path="/var/lib/kubelet/pods/2f090d42-ab12-4e84-af58-5370016c2733/volumes" Oct 30 22:16:46.871869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:46.871845 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25cea18-75f3-443d-957d-7d32ff2a1322" path="/var/lib/kubelet/pods/f25cea18-75f3-443d-957d-7d32ff2a1322/volumes" Oct 30 22:16:48.014860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.014824 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp"] Oct 30 22:16:48.015461 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015446 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2f090d42-ab12-4e84-af58-5370016c2733" containerName="payload-processing" Oct 30 22:16:48.015509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015463 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f090d42-ab12-4e84-af58-5370016c2733" containerName="payload-processing" Oct 30 22:16:48.015509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015489 5607 cpu_manager.go:414] "containerMap: removing container" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" Oct 30 22:16:48.015509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015495 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015512 5607 cpu_manager.go:414] "containerMap: removing container" podUID="38fc515f-55f6-49ac-815d-6b20ab486861" containerName="payload-pre-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015518 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fc515f-55f6-49ac-815d-6b20ab486861" containerName="payload-pre-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015536 5607 cpu_manager.go:414] "containerMap: removing container" podUID="ea75a11f-47ab-414b-b350-9e870be08c99" containerName="payload-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015540 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea75a11f-47ab-414b-b350-9e870be08c99" containerName="payload-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015546 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f25cea18-75f3-443d-957d-7d32ff2a1322" containerName="payload-pre-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015550 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25cea18-75f3-443d-957d-7d32ff2a1322" containerName="payload-pre-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015560 5607 cpu_manager.go:414] "containerMap: removing container" podUID="ce3b65ac-f714-4bd1-9c7e-7feefc36316b" containerName="payload-pre-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015565 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3b65ac-f714-4bd1-9c7e-7feefc36316b" containerName="payload-pre-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015579 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a60bc0c4-0d00-40c5-a610-0e247043568d" containerName="payload-processing" Oct 30 22:16:48.015599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015583 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60bc0c4-0d00-40c5-a610-0e247043568d" containerName="payload-processing" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015667 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60bc0c4-0d00-40c5-a610-0e247043568d" containerName="payload-processing" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015676 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f090d42-ab12-4e84-af58-5370016c2733" containerName="payload-processing" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015684 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="212ef815-88f1-42d0-820d-d20ff273e1a6" containerName="istio-proxy" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015692 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fc515f-55f6-49ac-815d-6b20ab486861" containerName="payload-pre-processing" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015701 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea75a11f-47ab-414b-b350-9e870be08c99" containerName="payload-processing" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015707 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25cea18-75f3-443d-957d-7d32ff2a1322" containerName="payload-pre-processing" Oct 30 22:16:48.015886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.015712 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3b65ac-f714-4bd1-9c7e-7feefc36316b" containerName="payload-pre-processing" Oct 30 22:16:48.021126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.021108 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.072666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.072634 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcqc\" (UniqueName: \"kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.072838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.072705 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.078358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.078335 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp"] Oct 30 22:16:48.125766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.125735 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4576d9\"" Oct 30 22:16:48.125766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.125742 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4576d9-dockercfg-d54zk\"" Oct 30 22:16:48.182556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.182520 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt"] Oct 30 22:16:48.205841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.202529 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.249040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.249001 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcqc\" (UniqueName: \"kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.250449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.209679 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.256587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.256245 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt"] Oct 30 22:16:48.258529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.258251 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.312381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.312349 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcqc\" (UniqueName: \"kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc\") pod \"payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.334725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.334692 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:48.350424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.350396 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbw2\" (UniqueName: \"kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2\") pod \"payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.350542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.350444 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.452549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.452511 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbw2\" (UniqueName: \"kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2\") pod \"payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.452700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.452587 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.454177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.453806 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.517542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.517509 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbw2\" (UniqueName: \"kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2\") pod \"payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.599721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.599637 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:48.723184 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.723154 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp"] Oct 30 22:16:48.875327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.875299 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" event={"ID":"c4655842-8ff9-49fb-af54-12c7972a8f4c","Type":"ContainerStarted","Data":"8b3124586c1c4609518f440bb1d3ff75d9a772821a1a08dfbbabf72610ac5dcb"} Oct 30 22:16:48.981447 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:16:48.981410 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c0067c_db3f_4662_8e4d_ee2f868271cf.slice/crio-7cbaa55765d6b28962769bd7c74f8db7773b34fa1db3d3646092fc40d31ca1d7 WatchSource:0}: Error finding container 7cbaa55765d6b28962769bd7c74f8db7773b34fa1db3d3646092fc40d31ca1d7: Status 404 returned error can't find the container with id 7cbaa55765d6b28962769bd7c74f8db7773b34fa1db3d3646092fc40d31ca1d7 Oct 30 22:16:48.984065 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:48.984040 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt"] Oct 30 22:16:49.888114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:49.888073 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" event={"ID":"c4655842-8ff9-49fb-af54-12c7972a8f4c","Type":"ContainerStarted","Data":"0a58c35356491978d06c2ced6c413741ce63cbd451d40142ff73b46fd5a1f2e9"} Oct 30 22:16:49.888610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:49.888301 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:16:49.889941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:49.889915 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" event={"ID":"f7c0067c-db3f-4662-8e4d-ee2f868271cf","Type":"ContainerStarted","Data":"f6fadd718f74b5a8f4e72fbe5ae6d27fc6a7476b97328b8d90d18dde1a4eac2a"} Oct 30 22:16:49.890078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:49.889982 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" event={"ID":"f7c0067c-db3f-4662-8e4d-ee2f868271cf","Type":"ContainerStarted","Data":"7cbaa55765d6b28962769bd7c74f8db7773b34fa1db3d3646092fc40d31ca1d7"} Oct 30 22:16:49.890078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:49.890014 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:16:50.008199 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:50.008150 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" podStartSLOduration=3.008136147 podStartE2EDuration="3.008136147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.006664284 +0000 UTC m=+1789.407220406" watchObservedRunningTime="2026-10-30 22:16:50.008136147 +0000 UTC m=+1789.408692267" Oct 30 22:16:50.126307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:50.126248 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" podStartSLOduration=2.126230433 podStartE2EDuration="2.126230433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.123989707 +0000 UTC m=+1789.524545836" watchObservedRunningTime="2026-10-30 22:16:50.126230433 +0000 UTC m=+1789.526786642" Oct 30 22:16:52.865053 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:52.865019 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:16:54.565346 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.565315 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq"] Oct 30 22:16:54.569850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.569827 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.625700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.625678 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8b4b54-serving-cert\"" Oct 30 22:16:54.679644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.679615 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq"] Oct 30 22:16:54.736617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.736591 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.736763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.736628 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gr4\" (UniqueName: \"kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.736835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.736763 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.736835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.736829 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.839877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.839779 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.839877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.839846 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gr4\" (UniqueName: \"kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.840523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.839996 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.840523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.840077 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.841099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.841072 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.845089 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:54.842049 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-serving-cert: secret "maas-api-e2e-ipp-b-8b4b54-serving-cert" not found Oct 30 22:16:54.845089 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:16:54.842159 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls podName:60619e7b-3510-4efb-b180-49f7804ce2af nodeName:}" failed. No retries permitted until 2026-10-30 22:16:55.342131288 +0000 UTC m=+1794.742687441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls") pod "maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" (UID: "60619e7b-3510-4efb-b180-49f7804ce2af") : secret "maas-api-e2e-ipp-b-8b4b54-serving-cert" not found Oct 30 22:16:54.856744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.856712 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:54.904651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:54.904624 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gr4\" (UniqueName: \"kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:55.347015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:55.346986 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:55.349384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:55.349362 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:55.484668 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:55.484632 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:55.864977 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:16:55.864923 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60619e7b_3510_4efb_b180_49f7804ce2af.slice/crio-7b348a4fbcc5760acbb8d47d03ce4c99681d02cb9e8d9e055dc2feee8237cf65 WatchSource:0}: Error finding container 7b348a4fbcc5760acbb8d47d03ce4c99681d02cb9e8d9e055dc2feee8237cf65: Status 404 returned error can't find the container with id 7b348a4fbcc5760acbb8d47d03ce4c99681d02cb9e8d9e055dc2feee8237cf65 Oct 30 22:16:55.868314 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:55.868290 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq"] Oct 30 22:16:55.930013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:55.929982 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" event={"ID":"60619e7b-3510-4efb-b180-49f7804ce2af","Type":"ContainerStarted","Data":"7b348a4fbcc5760acbb8d47d03ce4c99681d02cb9e8d9e055dc2feee8237cf65"} Oct 30 22:16:56.937778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:56.937745 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" event={"ID":"60619e7b-3510-4efb-b180-49f7804ce2af","Type":"ContainerStarted","Data":"1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2"} Oct 30 22:16:56.938162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:56.937867 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:16:57.084145 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:16:57.084058 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" podStartSLOduration=2.129923986 podStartE2EDuration="3.084045045s" totalImagesPullingTime="954.121059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:57.083242837 +0000 UTC m=+1796.483798957" watchObservedRunningTime="2026-10-30 22:16:57.084045045 +0000 UTC m=+1796.484601212" Oct 30 22:17:00.899193 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:00.899164 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:17:00.899583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:00.899352 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:17:01.118879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:01.118851 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:17:01.121392 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:01.121372 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:17:02.950688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:02.950663 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:17:03.480188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.480156 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr"] Oct 30 22:17:03.480405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.480383 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" podUID="a861c63d-9146-47ec-b1d3-9bb44de10feb" containerName="maas-api" containerID="cri-o://dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6" gracePeriod=30 Oct 30 22:17:03.776331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.776306 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:17:03.851453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.851417 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca\") pod \"a861c63d-9146-47ec-b1d3-9bb44de10feb\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " Oct 30 22:17:03.851842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.851820 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp\" (UniqueName: \"kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp\") pod \"a861c63d-9146-47ec-b1d3-9bb44de10feb\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " Oct 30 22:17:03.852041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.852002 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs\") pod \"a861c63d-9146-47ec-b1d3-9bb44de10feb\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " Oct 30 22:17:03.852310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.852294 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\") pod \"a861c63d-9146-47ec-b1d3-9bb44de10feb\" (UID: \"a861c63d-9146-47ec-b1d3-9bb44de10feb\") " Oct 30 22:17:03.854188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.854142 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca" pod "a861c63d-9146-47ec-b1d3-9bb44de10feb" (UID: "a861c63d-9146-47ec-b1d3-9bb44de10feb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:03.857286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.856857 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls" pod "a861c63d-9146-47ec-b1d3-9bb44de10feb" (UID: "a861c63d-9146-47ec-b1d3-9bb44de10feb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:03.859710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.859686 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs" pod "a861c63d-9146-47ec-b1d3-9bb44de10feb" (UID: "a861c63d-9146-47ec-b1d3-9bb44de10feb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:03.860264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.860243 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp" pod "a861c63d-9146-47ec-b1d3-9bb44de10feb" (UID: "a861c63d-9146-47ec-b1d3-9bb44de10feb"). InnerVolumeSpecName "kube-api-access-t7swp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:03.955141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.955105 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7swp\" (UniqueName: \"kubernetes.io/projected/a861c63d-9146-47ec-b1d3-9bb44de10feb-kube-api-access-t7swp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.955141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.955140 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.955700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.955155 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a861c63d-9146-47ec-b1d3-9bb44de10feb-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.955700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.955170 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a861c63d-9146-47ec-b1d3-9bb44de10feb-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:03.983103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.983073 5607 generic.go:356] "Generic (PLEG): container finished" podID="a861c63d-9146-47ec-b1d3-9bb44de10feb" containerID="dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6" exitCode=0 Oct 30 22:17:03.983245 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.983125 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" Oct 30 22:17:03.983245 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.983141 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" event={"ID":"a861c63d-9146-47ec-b1d3-9bb44de10feb","Type":"ContainerDied","Data":"dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6"} Oct 30 22:17:03.983245 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.983195 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr" event={"ID":"a861c63d-9146-47ec-b1d3-9bb44de10feb","Type":"ContainerDied","Data":"b191a6ed582b4324ee0b77b475ad05a251fd66eacf276d46504dd3dda4c57d21"} Oct 30 22:17:03.983245 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.983222 5607 scope.go:122] "RemoveContainer" containerID="dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6" Oct 30 22:17:03.994506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.994489 5607 scope.go:122] "RemoveContainer" containerID="dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6" Oct 30 22:17:03.994752 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:03.994737 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6\": container with ID starting with dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6 not found: ID does not exist" containerID="dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6" Oct 30 22:17:03.994794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:03.994760 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6"} err="failed to get container status \"dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6\": rpc error: code = NotFound desc = could not find container \"dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6\": container with ID starting with dc54b71a683a00f964dc9fe03e5a33cbf0b9abf6a023b47131f943f0b17334e6 not found: ID does not exist" Oct 30 22:17:04.152169 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:04.152130 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr"] Oct 30 22:17:04.155995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:04.155934 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4401bb-57d7bb5b75-p6jnr"] Oct 30 22:17:04.871507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:04.871476 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a861c63d-9146-47ec-b1d3-9bb44de10feb" path="/var/lib/kubelet/pods/a861c63d-9146-47ec-b1d3-9bb44de10feb/volumes" Oct 30 22:17:12.377571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:12.377494 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx"] Oct 30 22:17:12.378001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:12.377712 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" containerID="cri-o://57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f" gracePeriod=30 Oct 30 22:17:13.931738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:13.931701 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn"] Oct 30 22:17:13.932430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:13.932412 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a861c63d-9146-47ec-b1d3-9bb44de10feb" containerName="maas-api" Oct 30 22:17:13.932494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:13.932430 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a861c63d-9146-47ec-b1d3-9bb44de10feb" containerName="maas-api" Oct 30 22:17:13.932548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:13.932524 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a861c63d-9146-47ec-b1d3-9bb44de10feb" containerName="maas-api" Oct 30 22:17:13.937438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:13.937416 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.001863 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:13.996357 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8daffd-openshift-default-dockercfg-7pqls\"" Oct 30 22:17:14.060359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.060327 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn"] Oct 30 22:17:14.096070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096035 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6cf\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096105 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096199 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096243 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096400 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096266 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096415 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096457 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096488 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.096665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.096645 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.197924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.197892 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6cf\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.197971 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198033 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198190 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198244 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198339 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198384 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198426 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198517 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198582 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198595 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198710 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.198781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.198769 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.199215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.199192 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.200849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.200819 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.201040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.201020 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.263494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.263462 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.263729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.263701 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6cf\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf\") pod \"e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.558266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.558162 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:14.987825 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:17:14.987785 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42d7934_e97a_45fd_a4b1_58420951e26e.slice/crio-5a538cafa29adf27f41a2d44854529cca63dbdbaf12b334fb94abdb19644b736 WatchSource:0}: Error finding container 5a538cafa29adf27f41a2d44854529cca63dbdbaf12b334fb94abdb19644b736: Status 404 returned error can't find the container with id 5a538cafa29adf27f41a2d44854529cca63dbdbaf12b334fb94abdb19644b736 Oct 30 22:17:14.990411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.990377 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:17:14.990508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.990456 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:17:14.990508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.990485 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:17:14.991031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:14.990928 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn"] Oct 30 22:17:15.073187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:15.073156 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" event={"ID":"d42d7934-e97a-45fd-a4b1-58420951e26e","Type":"ContainerStarted","Data":"ea4c355f3383c81122f9d1f51d6fd01ef7b690de715fd8a10861f9471dbf829a"} Oct 30 22:17:15.073328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:15.073251 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" event={"ID":"d42d7934-e97a-45fd-a4b1-58420951e26e","Type":"ContainerStarted","Data":"5a538cafa29adf27f41a2d44854529cca63dbdbaf12b334fb94abdb19644b736"} Oct 30 22:17:15.197684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:15.197635 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podStartSLOduration=2.197619587 podStartE2EDuration="2.197619587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:15.195439057 +0000 UTC m=+1814.595995203" watchObservedRunningTime="2026-10-30 22:17:15.197619587 +0000 UTC m=+1814.598175707" Oct 30 22:17:15.558775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:15.558746 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:16.563610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:16.563581 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:17.085652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.085622 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:17.086409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.086390 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:17:17.767097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.767073 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:17:17.850170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850142 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850199 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850252 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850330 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850356 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850404 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850430 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850734 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850511 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850734 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850569 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs\") pod \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\" (UID: \"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b\") " Oct 30 22:17:17.850734 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850428 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:17.850734 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850591 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:17.850734 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850696 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:17.851038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.850791 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:17.851038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.851026 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:17.851561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.851536 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.851633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.851566 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.851633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.851584 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.851633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.851599 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.851633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.851614 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.852993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.852831 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:17.852993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.852882 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:17.853219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.853199 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:17.853292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.853232 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g" pod "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" (UID: "f18c2dc3-2e1a-4647-9b5d-77c6d383f25b"). InnerVolumeSpecName "kube-api-access-9rv4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:17.952905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.952883 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.952905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.952906 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.953163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.952917 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rv4g\" (UniqueName: \"kubernetes.io/projected/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-kube-api-access-9rv4g\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.953163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:17.952927 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:18.094732 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.094703 5607 generic.go:356] "Generic (PLEG): container finished" podID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerID="57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f" exitCode=0 Oct 30 22:17:18.094939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.094752 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" Oct 30 22:17:18.094939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.094781 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" event={"ID":"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b","Type":"ContainerDied","Data":"57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f"} Oct 30 22:17:18.094939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.094840 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx" event={"ID":"f18c2dc3-2e1a-4647-9b5d-77c6d383f25b","Type":"ContainerDied","Data":"6291f2ab3e1858fd50d3382435d711ed1344940abed08d0cdb5c7a19b6f146cb"} Oct 30 22:17:18.094939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.094870 5607 scope.go:122] "RemoveContainer" containerID="57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f" Oct 30 22:17:18.106135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.106117 5607 scope.go:122] "RemoveContainer" containerID="90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b" Oct 30 22:17:18.114998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.114982 5607 scope.go:122] "RemoveContainer" containerID="57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f" Oct 30 22:17:18.115213 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:18.115197 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f\": container with ID starting with 57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f not found: ID does not exist" containerID="57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f" Oct 30 22:17:18.115260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.115223 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f"} err="failed to get container status \"57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f\": rpc error: code = NotFound desc = could not find container \"57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f\": container with ID starting with 57b6c65453636cb1673d962d0701d5a40fb703f930746c56e61f55bf67712b3f not found: ID does not exist" Oct 30 22:17:18.115260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.115239 5607 scope.go:122] "RemoveContainer" containerID="90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b" Oct 30 22:17:18.115429 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:18.115413 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b\": container with ID starting with 90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b not found: ID does not exist" containerID="90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b" Oct 30 22:17:18.115469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.115434 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b"} err="failed to get container status \"90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b\": rpc error: code = NotFound desc = could not find container \"90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b\": container with ID starting with 90ac4b9ad07055b9d3adf3edf09f819e34ac65fd06d20a79f9ada585406d5e3b not found: ID does not exist" Oct 30 22:17:18.264692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.264642 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx"] Oct 30 22:17:18.268974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.268926 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4401bb-openshift-default-57d6f47c85-8vqcx"] Oct 30 22:17:18.877943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:18.877910 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" path="/var/lib/kubelet/pods/f18c2dc3-2e1a-4647-9b5d-77c6d383f25b/volumes" Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.286360 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z"] Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.287490 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.287509 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.287526 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.287534 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.287698 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" Oct 30 22:17:22.287978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.287711 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18c2dc3-2e1a-4647-9b5d-77c6d383f25b" containerName="istio-proxy" Oct 30 22:17:22.306313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.306284 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.321062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.321021 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7z9s\" (UniqueName: \"kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s\") pod \"payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.321449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.321427 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.383484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.383445 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z"] Oct 30 22:17:22.396136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.396101 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8daffd\"" Oct 30 22:17:22.396136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.396110 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8daffd-dockercfg-tvvxj\"" Oct 30 22:17:22.423258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.423234 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.423374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.423357 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7z9s\" (UniqueName: \"kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s\") pod \"payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.423906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.423887 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.449865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.449842 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8"] Oct 30 22:17:22.454081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.454063 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.486180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.486156 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7z9s\" (UniqueName: \"kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s\") pod \"payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.509513 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.509487 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8"] Oct 30 22:17:22.524481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.524456 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgnv\" (UniqueName: \"kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.524585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.524503 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.625894 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.625823 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgnv\" (UniqueName: \"kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.625894 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.625861 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.626577 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.626558 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.684925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.684904 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:22.688298 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.688172 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgnv\" (UniqueName: \"kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.768345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:22.768320 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:22.872002 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:22.871701 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:17:23.057431 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:17:23.057400 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe218379_79d6_4eb1_81f9_989b79f45d4e.slice/crio-3ad0330a2d78b6a561157264f97f71915ffc9cff0bbc00f5c6953061b17543e4 WatchSource:0}: Error finding container 3ad0330a2d78b6a561157264f97f71915ffc9cff0bbc00f5c6953061b17543e4: Status 404 returned error can't find the container with id 3ad0330a2d78b6a561157264f97f71915ffc9cff0bbc00f5c6953061b17543e4 Oct 30 22:17:23.059589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:23.059566 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z"] Oct 30 22:17:23.130405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:23.130380 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" event={"ID":"fe218379-79d6-4eb1-81f9-989b79f45d4e","Type":"ContainerStarted","Data":"3ad0330a2d78b6a561157264f97f71915ffc9cff0bbc00f5c6953061b17543e4"} Oct 30 22:17:23.160860 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:17:23.160825 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909229f2_f9dd_4759_84f7_5f62f4e61676.slice/crio-0dbbe5d117ee72c2c4d12a6f74f4f671777fff6138eaf9cd002d86b0b75bcf6f WatchSource:0}: Error finding container 0dbbe5d117ee72c2c4d12a6f74f4f671777fff6138eaf9cd002d86b0b75bcf6f: Status 404 returned error can't find the container with id 0dbbe5d117ee72c2c4d12a6f74f4f671777fff6138eaf9cd002d86b0b75bcf6f Oct 30 22:17:23.165050 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:23.165028 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8"] Oct 30 22:17:24.163439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.163399 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" event={"ID":"909229f2-f9dd-4759-84f7-5f62f4e61676","Type":"ContainerStarted","Data":"1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3"} Oct 30 22:17:24.164022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.163469 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" event={"ID":"909229f2-f9dd-4759-84f7-5f62f4e61676","Type":"ContainerStarted","Data":"0dbbe5d117ee72c2c4d12a6f74f4f671777fff6138eaf9cd002d86b0b75bcf6f"} Oct 30 22:17:24.164423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.164401 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:24.166864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.166834 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" event={"ID":"fe218379-79d6-4eb1-81f9-989b79f45d4e","Type":"ContainerStarted","Data":"952abd95b0805151876940fee7f7b68068b6825bef9a58cedb4e2f24a305a3cd"} Oct 30 22:17:24.167282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.167261 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:24.284073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.283923 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" podStartSLOduration=2.28390781 podStartE2EDuration="2.28390781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:24.283024484 +0000 UTC m=+1823.683580606" watchObservedRunningTime="2026-10-30 22:17:24.28390781 +0000 UTC m=+1823.684463931" Oct 30 22:17:24.404138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:24.404077 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" podStartSLOduration=2.404063153 podStartE2EDuration="2.404063153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:24.400635928 +0000 UTC m=+1823.801192050" watchObservedRunningTime="2026-10-30 22:17:24.404063153 +0000 UTC m=+1823.804619296" Oct 30 22:17:25.695982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.695939 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr"] Oct 30 22:17:25.700261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.700246 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:25.761016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.760990 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr"] Oct 30 22:17:25.784005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.783927 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:25.784211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.784184 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28qh\" (UniqueName: \"kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh\") pod \"payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:25.810702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.807102 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ad7da0\"" Oct 30 22:17:25.810702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.807261 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ad7da0-dockercfg-ngcns\"" Oct 30 22:17:25.885993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.885939 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:25.886135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.886058 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28qh\" (UniqueName: \"kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh\") pod \"payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:25.886621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.886600 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:25.948646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:25.948588 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28qh\" (UniqueName: \"kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh\") pod \"payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:26.015289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:26.015263 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:26.393450 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:17:26.393425 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126007cb_90a8_424f_a8a8_412ab2af1b6f.slice/crio-59f54ad72e430dbb900aa1bbbbb99f5f62d142d7933e0b03cdf6b14477935f7b WatchSource:0}: Error finding container 59f54ad72e430dbb900aa1bbbbb99f5f62d142d7933e0b03cdf6b14477935f7b: Status 404 returned error can't find the container with id 59f54ad72e430dbb900aa1bbbbb99f5f62d142d7933e0b03cdf6b14477935f7b Oct 30 22:17:26.397246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:26.397204 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr"] Oct 30 22:17:27.188291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:27.188251 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" event={"ID":"126007cb-90a8-424f-a8a8-412ab2af1b6f","Type":"ContainerStarted","Data":"47a7f46a66f54491e6c734623ecd6bfa73e436412af0d44fbb291fee0b6b28f9"} Oct 30 22:17:27.188777 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:27.188312 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" event={"ID":"126007cb-90a8-424f-a8a8-412ab2af1b6f","Type":"ContainerStarted","Data":"59f54ad72e430dbb900aa1bbbbb99f5f62d142d7933e0b03cdf6b14477935f7b"} Oct 30 22:17:27.188777 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:27.188347 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:27.309406 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:27.309364 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" podStartSLOduration=2.309352398 podStartE2EDuration="2.309352398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:27.307834659 +0000 UTC m=+1826.708390782" watchObservedRunningTime="2026-10-30 22:17:27.309352398 +0000 UTC m=+1826.709908519" Oct 30 22:17:35.175714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:35.175686 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:17:36.182244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:36.182218 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:17:38.199073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:38.199045 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:17:39.803002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:39.802938 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9"] Oct 30 22:17:39.803500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:39.803199 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" containerID="cri-o://0aaefa7ebd87be897a41007b90713ab363b02243b795419972ac1970c2b3d7a5" gracePeriod=30 Oct 30 22:17:39.822244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:39.822210 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h"] Oct 30 22:17:39.822444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:39.822423 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" containerID="cri-o://e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62" gracePeriod=30 Oct 30 22:17:39.841697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:39.841668 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74"] Oct 30 22:17:39.841857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:39.841838 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" containerID="cri-o://eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07" gracePeriod=30 Oct 30 22:17:45.131063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.131033 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:17:45.157766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.157739 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:17:45.221960 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.221935 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.222078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.221979 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.221998 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222057 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222092 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.222254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222114 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.222254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222160 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.222254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222202 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.222254 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222228 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222261 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.222520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222258 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.222520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222292 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222347 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222376 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.222520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222376 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.222838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222521 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.222838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222548 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.222838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222737 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.222838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222745 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.223261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222912 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:45.223261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.222993 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.223261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.223035 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.223261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.223078 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert\") pod \"2d2baf35-6241-446f-9963-61b6702f46ef\" (UID: \"2d2baf35-6241-446f-9963-61b6702f46ef\") " Oct 30 22:17:45.223261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.223126 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.223261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.223178 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data\") pod \"60ee2745-6993-4e15-911b-bcee91b54dc5\" (UID: \"60ee2745-6993-4e15-911b-bcee91b54dc5\") " Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.223870 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224100 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224122 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224138 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224152 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224166 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224180 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224194 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224209 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ee2745-6993-4e15-911b-bcee91b54dc5-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.224602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224430 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.225282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224609 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:45.225282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224927 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.225282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.224995 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.225282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.225205 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:45.225447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.225430 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "kube-api-access-ks5xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.225538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.225517 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:45.225699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.225678 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "kube-api-access-v6xhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.226398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.226383 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token" pod "60ee2745-6993-4e15-911b-bcee91b54dc5" (UID: "60ee2745-6993-4e15-911b-bcee91b54dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.226487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.226474 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy" pod "2d2baf35-6241-446f-9963-61b6702f46ef" (UID: "2d2baf35-6241-446f-9963-61b6702f46ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.312724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.312693 5607 generic.go:356] "Generic (PLEG): container finished" podID="2d2baf35-6241-446f-9963-61b6702f46ef" containerID="eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07" exitCode=0 Oct 30 22:17:45.312875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.312755 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" Oct 30 22:17:45.312875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.312770 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" event={"ID":"2d2baf35-6241-446f-9963-61b6702f46ef","Type":"ContainerDied","Data":"eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07"} Oct 30 22:17:45.312875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.312823 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74" event={"ID":"2d2baf35-6241-446f-9963-61b6702f46ef","Type":"ContainerDied","Data":"fbc08075b1b112cd9f678df1b96635bacb81c9c6dcfd1cd3845efc225f65ce24"} Oct 30 22:17:45.312875 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.312854 5607 scope.go:122] "RemoveContainer" containerID="eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07" Oct 30 22:17:45.315147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.315123 5607 generic.go:356] "Generic (PLEG): container finished" podID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerID="0aaefa7ebd87be897a41007b90713ab363b02243b795419972ac1970c2b3d7a5" exitCode=0 Oct 30 22:17:45.315265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.315203 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" event={"ID":"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad","Type":"ContainerDied","Data":"0aaefa7ebd87be897a41007b90713ab363b02243b795419972ac1970c2b3d7a5"} Oct 30 22:17:45.317431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.317411 5607 generic.go:356] "Generic (PLEG): container finished" podID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerID="e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62" exitCode=0 Oct 30 22:17:45.317769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.317679 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" Oct 30 22:17:45.318054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.317697 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" event={"ID":"60ee2745-6993-4e15-911b-bcee91b54dc5","Type":"ContainerDied","Data":"e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62"} Oct 30 22:17:45.318054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.318019 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h" event={"ID":"60ee2745-6993-4e15-911b-bcee91b54dc5","Type":"ContainerDied","Data":"dded8a5b2c0e59b819383ca12a686888c314f01d1885775f794938381e5781c7"} Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325427 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325452 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325465 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xhw\" (UniqueName: \"kubernetes.io/projected/2d2baf35-6241-446f-9963-61b6702f46ef-kube-api-access-v6xhw\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325481 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2baf35-6241-446f-9963-61b6702f46ef-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325495 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5xk\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-kube-api-access-ks5xk\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325508 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325521 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2baf35-6241-446f-9963-61b6702f46ef-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325533 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2baf35-6241-446f-9963-61b6702f46ef-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325545 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.327048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.325558 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ee2745-6993-4e15-911b-bcee91b54dc5-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.328601 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.328495 5607 scope.go:122] "RemoveContainer" containerID="eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07" Oct 30 22:17:45.328896 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:45.328861 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07\": container with ID starting with eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07 not found: ID does not exist" containerID="eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07" Oct 30 22:17:45.329002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.328901 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07"} err="failed to get container status \"eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07\": rpc error: code = NotFound desc = could not find container \"eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07\": container with ID starting with eff04cc1b927ce5aa344b69bf9d26571bd51dde1180c15a0e980ececcf79ff07 not found: ID does not exist" Oct 30 22:17:45.329002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.328924 5607 scope.go:122] "RemoveContainer" containerID="e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62" Oct 30 22:17:45.342396 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.342378 5607 scope.go:122] "RemoveContainer" containerID="e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62" Oct 30 22:17:45.342747 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:45.342720 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62\": container with ID starting with e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62 not found: ID does not exist" containerID="e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62" Oct 30 22:17:45.342835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.342755 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62"} err="failed to get container status \"e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62\": rpc error: code = NotFound desc = could not find container \"e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62\": container with ID starting with e710675f41992ae0f89eb8846f8a2af677a3379cec0a1ec5f22960bb7044da62 not found: ID does not exist" Oct 30 22:17:45.482588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.482562 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:17:45.527432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527400 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527450 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527510 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527554 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527608 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527651 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527717 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527776 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.527842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.527801 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo\") pod \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\" (UID: \"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad\") " Oct 30 22:17:45.529152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.529124 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.529359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.529334 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.529735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.529710 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.530822 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.530012 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:45.531868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.531820 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:45.531868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.531828 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.532330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.532309 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:45.532574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.532552 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "kube-api-access-q7m5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.533048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.533031 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token" pod "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" (UID: "55dd3a44-42b4-4aab-8b2c-f0b24067d2ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:45.533218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.533200 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74"] Oct 30 22:17:45.539749 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.539727 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-s2r74"] Oct 30 22:17:45.628995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.628965 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.628995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.628992 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629003 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629017 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629032 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7m5h\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-kube-api-access-q7m5h\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629041 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629050 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629058 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.629146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.629066 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:45.651119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.651057 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h"] Oct 30 22:17:45.657713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:45.657692 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-qr79h"] Oct 30 22:17:46.327812 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.327784 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" Oct 30 22:17:46.328316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.327825 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9" event={"ID":"55dd3a44-42b4-4aab-8b2c-f0b24067d2ad","Type":"ContainerDied","Data":"5f67b684e24d700b4b3c6b8025c5d5ed49c72b371b12f63ab018c0532b032cf9"} Oct 30 22:17:46.328316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.327886 5607 scope.go:122] "RemoveContainer" containerID="0aaefa7ebd87be897a41007b90713ab363b02243b795419972ac1970c2b3d7a5" Oct 30 22:17:46.513045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.513007 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9"] Oct 30 22:17:46.517628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.517598 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-mzqb9"] Oct 30 22:17:46.870790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.870755 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" path="/var/lib/kubelet/pods/2d2baf35-6241-446f-9963-61b6702f46ef/volumes" Oct 30 22:17:46.871246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.871232 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" path="/var/lib/kubelet/pods/55dd3a44-42b4-4aab-8b2c-f0b24067d2ad/volumes" Oct 30 22:17:46.871641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:46.871629 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" path="/var/lib/kubelet/pods/60ee2745-6993-4e15-911b-bcee91b54dc5/volumes" Oct 30 22:17:52.864505 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:17:52.864466 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:17:55.363017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:55.362984 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw"] Oct 30 22:17:55.473650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:17:55.473617 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm"] Oct 30 22:18:00.363817 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.363752 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" podUID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" containerName="payload-processing" containerID="cri-o://b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b" gracePeriod=25 Oct 30 22:18:00.474025 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.473917 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" podUID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" containerName="payload-pre-processing" containerID="cri-o://40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3" gracePeriod=25 Oct 30 22:18:00.626104 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:18:00.626027 5607 logging.go:55] [core] [Channel #4848 SubChannel #4850]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.150:50052", ServerName: "10.133.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.150:50052: connect: connection refused" Oct 30 22:18:00.698617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.698590 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:18:00.767737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.767692 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw"] Oct 30 22:18:00.767991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.767932 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" containerID="cri-o://0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5" gracePeriod=30 Oct 30 22:18:00.815710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.815686 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:18:00.861033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.858457 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume\") pod \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " Oct 30 22:18:00.861033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.858708 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br\" (UniqueName: \"kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br\") pod \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\" (UID: \"a5d2487b-1050-4d75-9c7a-ed6ae2114e70\") " Oct 30 22:18:00.888062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.882906 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br" pod "a5d2487b-1050-4d75-9c7a-ed6ae2114e70" (UID: "a5d2487b-1050-4d75-9c7a-ed6ae2114e70"). InnerVolumeSpecName "kube-api-access-lr2br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:00.891345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.891308 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume" pod "a5d2487b-1050-4d75-9c7a-ed6ae2114e70" (UID: "a5d2487b-1050-4d75-9c7a-ed6ae2114e70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:00.960062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.960025 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn\" (UniqueName: \"kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn\") pod \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " Oct 30 22:18:00.960204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.960080 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume\") pod \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\" (UID: \"8314d6c5-c110-4855-8ca6-f101d7b8a9f3\") " Oct 30 22:18:00.960470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.960449 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume" pod "8314d6c5-c110-4855-8ca6-f101d7b8a9f3" (UID: "8314d6c5-c110-4855-8ca6-f101d7b8a9f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:00.960841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.960823 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:00.960841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.960841 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2br\" (UniqueName: \"kubernetes.io/projected/a5d2487b-1050-4d75-9c7a-ed6ae2114e70-kube-api-access-lr2br\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:00.961043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.960855 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:00.962686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:00.962660 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn" pod "8314d6c5-c110-4855-8ca6-f101d7b8a9f3" (UID: "8314d6c5-c110-4855-8ca6-f101d7b8a9f3"). InnerVolumeSpecName "kube-api-access-2nqhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:01.062512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.062482 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqhn\" (UniqueName: \"kubernetes.io/projected/8314d6c5-c110-4855-8ca6-f101d7b8a9f3-kube-api-access-2nqhn\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:01.432105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.432070 5607 generic.go:356] "Generic (PLEG): container finished" podID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" containerID="b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b" exitCode=0 Oct 30 22:18:01.432448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.432142 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" Oct 30 22:18:01.432448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.432138 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" event={"ID":"a5d2487b-1050-4d75-9c7a-ed6ae2114e70","Type":"ContainerDied","Data":"b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b"} Oct 30 22:18:01.432448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.432333 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" event={"ID":"a5d2487b-1050-4d75-9c7a-ed6ae2114e70","Type":"ContainerDied","Data":"1a12a7b6dea479ff0e94e61d7d4bc625a849b69853a73be7926583ce85fd6cef"} Oct 30 22:18:01.432448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.432362 5607 scope.go:122] "RemoveContainer" containerID="b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b" Oct 30 22:18:01.433746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.433651 5607 generic.go:356] "Generic (PLEG): container finished" podID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" containerID="40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3" exitCode=0 Oct 30 22:18:01.433746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.433695 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" event={"ID":"8314d6c5-c110-4855-8ca6-f101d7b8a9f3","Type":"ContainerDied","Data":"40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3"} Oct 30 22:18:01.433746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.433711 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" Oct 30 22:18:01.433746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.433726 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" event={"ID":"8314d6c5-c110-4855-8ca6-f101d7b8a9f3","Type":"ContainerDied","Data":"9a956bb27bb40da0ddcc354765cc1137568e6bf902d2331825ccf66e913c2cdb"} Oct 30 22:18:01.444173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.444119 5607 scope.go:122] "RemoveContainer" containerID="b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b" Oct 30 22:18:01.444467 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:18:01.444440 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b\": container with ID starting with b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b not found: ID does not exist" containerID="b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b" Oct 30 22:18:01.444562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.444470 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b"} err="failed to get container status \"b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b\": rpc error: code = NotFound desc = could not find container \"b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b\": container with ID starting with b403b3588e402569dbbed891d21a40367c79deefc35e0054b689001dfd0ec55b not found: ID does not exist" Oct 30 22:18:01.444562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.444488 5607 scope.go:122] "RemoveContainer" containerID="40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3" Oct 30 22:18:01.453265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.453219 5607 scope.go:122] "RemoveContainer" containerID="40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3" Oct 30 22:18:01.453515 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:18:01.453496 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3\": container with ID starting with 40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3 not found: ID does not exist" containerID="40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3" Oct 30 22:18:01.453578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.453528 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3"} err="failed to get container status \"40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3\": rpc error: code = NotFound desc = could not find container \"40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3\": container with ID starting with 40ec25a31f7aff7c7e4be9b6b16c59f8b4bc28e17bd6888da7a4f5d5d68fc7e3 not found: ID does not exist" Oct 30 22:18:01.606099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.606074 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw"] Oct 30 22:18:01.614758 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.614737 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw"] Oct 30 22:18:01.625975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.625935 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm" podUID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.150:50052\" within 1s: context deadline exceeded" Oct 30 22:18:01.626059 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:18:01.626010 5607 logging.go:55] [core] [Channel #4847 SubChannel #4849]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.151:50052", ServerName: "10.133.0.151:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.151:50052: operation was canceled" Oct 30 22:18:01.626059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.625935 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-86c94b4997-ckdpw" podUID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.151:50052\" within 1s: context deadline exceeded" Oct 30 22:18:01.729789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.729766 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm"] Oct 30 22:18:01.735420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:01.735401 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-666c85768c-5gprm"] Oct 30 22:18:02.411513 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.411472 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq"] Oct 30 22:18:02.412175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412156 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" Oct 30 22:18:02.412175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412175 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412195 5607 cpu_manager.go:414] "containerMap: removing container" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412202 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412215 5607 cpu_manager.go:414] "containerMap: removing container" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412222 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412282 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" containerName="payload-processing" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412289 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" containerName="payload-processing" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412309 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" containerName="payload-pre-processing" Oct 30 22:18:02.412350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412317 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" containerName="payload-pre-processing" Oct 30 22:18:02.412728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412460 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" containerName="payload-pre-processing" Oct 30 22:18:02.412728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412478 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2baf35-6241-446f-9963-61b6702f46ef" containerName="istio-proxy" Oct 30 22:18:02.412728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412489 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ee2745-6993-4e15-911b-bcee91b54dc5" containerName="istio-proxy" Oct 30 22:18:02.412728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412500 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="55dd3a44-42b4-4aab-8b2c-f0b24067d2ad" containerName="istio-proxy" Oct 30 22:18:02.412728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.412518 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" containerName="payload-processing" Oct 30 22:18:02.417843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.417820 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.492068 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.492037 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0e90fe-openshift-default-dockercfg-kmlt9\"" Oct 30 22:18:02.545522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.545493 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq"] Oct 30 22:18:02.582627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.582594 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844ls\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.582815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.582645 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.582815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.582663 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.583001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.582810 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.583001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.582927 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.583102 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.583073 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.583151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.583131 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.583217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.583197 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.583325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.583308 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.684967 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844ls\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685034 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685066 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685251 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685356 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685455 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685504 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.685576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685532 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.687051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685616 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.687051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.685943 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.687051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.686170 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.687051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.686563 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.687051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.686754 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.687051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.687038 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.688382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.688340 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.689064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.689044 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.748469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.748439 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.748854 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.748832 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844ls\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls\") pod \"e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:02.872065 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.872031 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8314d6c5-c110-4855-8ca6-f101d7b8a9f3" path="/var/lib/kubelet/pods/8314d6c5-c110-4855-8ca6-f101d7b8a9f3/volumes" Oct 30 22:18:02.872485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:02.872471 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d2487b-1050-4d75-9c7a-ed6ae2114e70" path="/var/lib/kubelet/pods/a5d2487b-1050-4d75-9c7a-ed6ae2114e70/volumes" Oct 30 22:18:03.033883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:03.033807 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:03.507042 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:18:03.507006 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa5eb97_61b8_49de_8499_491918f935a7.slice/crio-8f2dbc0a1dc6d94310dd936ca0d3f73e6f1dc0a479630b21463ebf023ce7cc36 WatchSource:0}: Error finding container 8f2dbc0a1dc6d94310dd936ca0d3f73e6f1dc0a479630b21463ebf023ce7cc36: Status 404 returned error can't find the container with id 8f2dbc0a1dc6d94310dd936ca0d3f73e6f1dc0a479630b21463ebf023ce7cc36 Oct 30 22:18:03.509909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:03.509872 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:03.510015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:03.509982 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:03.510059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:03.510023 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:18:03.510836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:03.510811 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq"] Oct 30 22:18:03.536105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:03.536077 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" event={"ID":"2aa5eb97-61b8-49de-8499-491918f935a7","Type":"ContainerStarted","Data":"8f2dbc0a1dc6d94310dd936ca0d3f73e6f1dc0a479630b21463ebf023ce7cc36"} Oct 30 22:18:04.544109 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:04.544068 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" event={"ID":"2aa5eb97-61b8-49de-8499-491918f935a7","Type":"ContainerStarted","Data":"6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656"} Oct 30 22:18:04.775448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:04.775393 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podStartSLOduration=2.775378486 podStartE2EDuration="2.775378486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:04.722138859 +0000 UTC m=+1864.122694979" watchObservedRunningTime="2026-10-30 22:18:04.775378486 +0000 UTC m=+1864.175934610" Oct 30 22:18:05.034516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:05.034485 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:05.040160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:05.040135 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:05.551713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:05.551685 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:05.552663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:05.552642 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:18:06.042321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.042297 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:18:06.135836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.135804 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.135850 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.135997 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136036 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136071 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:06.136227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136171 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136323 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136400 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:06.136480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136419 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136451 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136533 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:06.136842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136542 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket\") pod \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\" (UID: \"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0\") " Oct 30 22:18:06.136842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136672 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:06.137115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.136971 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:06.137555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.137531 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.137650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.137561 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.137650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.137578 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.137650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.137592 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.137650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.137606 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.138576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.138543 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "kube-api-access-lfx22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:06.138683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.138644 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:06.138999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.138944 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:06.139106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.139009 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo" pod "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" (UID: "3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:06.238327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.238298 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.238327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.238323 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.238327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.238333 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.238580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.238342 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfx22\" (UniqueName: \"kubernetes.io/projected/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0-kube-api-access-lfx22\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:06.559167 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.559135 5607 generic.go:356] "Generic (PLEG): container finished" podID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerID="0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5" exitCode=0 Oct 30 22:18:06.559613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.559245 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" Oct 30 22:18:06.559613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.559238 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" event={"ID":"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0","Type":"ContainerDied","Data":"0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5"} Oct 30 22:18:06.559613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.559375 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw" event={"ID":"3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0","Type":"ContainerDied","Data":"b47785c56d42a331ecf54d1d927b3cbb1f98448b9aabe48033dc89be4c2c31c3"} Oct 30 22:18:06.559613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.559413 5607 scope.go:122] "RemoveContainer" containerID="0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5" Oct 30 22:18:06.570529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.570512 5607 scope.go:122] "RemoveContainer" containerID="baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252" Oct 30 22:18:06.579418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.579401 5607 scope.go:122] "RemoveContainer" containerID="0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5" Oct 30 22:18:06.579692 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:18:06.579673 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5\": container with ID starting with 0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5 not found: ID does not exist" containerID="0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5" Oct 30 22:18:06.579752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.579699 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5"} err="failed to get container status \"0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5\": rpc error: code = NotFound desc = could not find container \"0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5\": container with ID starting with 0b1964eeab79dd5ea6f4ecf55baff8ddc6df2cdd33935ebd9e4c7419bbf557b5 not found: ID does not exist" Oct 30 22:18:06.579752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.579715 5607 scope.go:122] "RemoveContainer" containerID="baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252" Oct 30 22:18:06.579943 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:18:06.579926 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252\": container with ID starting with baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252 not found: ID does not exist" containerID="baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252" Oct 30 22:18:06.580011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.579971 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252"} err="failed to get container status \"baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252\": rpc error: code = NotFound desc = could not find container \"baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252\": container with ID starting with baec5687a14a0fe6f6a3c00a1b32e1366aaada0ffd6f8324a7cd3fe2de0dc252 not found: ID does not exist" Oct 30 22:18:06.729723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.729700 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw"] Oct 30 22:18:06.780475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.780446 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-427d70-openshift-default-5b4c55bfcf-sw7lw"] Oct 30 22:18:06.870596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:06.870542 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" path="/var/lib/kubelet/pods/3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0/volumes" Oct 30 22:18:20.256978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:20.256095 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-gqbnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:20.256978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:20.256159 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:22.863244 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:18:22.863193 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:18:38.607075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:38.607042 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt"] Oct 30 22:18:38.719230 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:38.719199 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp"] Oct 30 22:18:43.607750 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.607666 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" podUID="f7c0067c-db3f-4662-8e4d-ee2f868271cf" containerName="payload-processing" containerID="cri-o://f6fadd718f74b5a8f4e72fbe5ae6d27fc6a7476b97328b8d90d18dde1a4eac2a" gracePeriod=25 Oct 30 22:18:43.720223 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.720182 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" podUID="c4655842-8ff9-49fb-af54-12c7972a8f4c" containerName="payload-pre-processing" containerID="cri-o://0a58c35356491978d06c2ced6c413741ce63cbd451d40142ff73b46fd5a1f2e9" gracePeriod=25 Oct 30 22:18:43.848760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.848730 5607 generic.go:356] "Generic (PLEG): container finished" podID="c4655842-8ff9-49fb-af54-12c7972a8f4c" containerID="0a58c35356491978d06c2ced6c413741ce63cbd451d40142ff73b46fd5a1f2e9" exitCode=0 Oct 30 22:18:43.848911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.848806 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" event={"ID":"c4655842-8ff9-49fb-af54-12c7972a8f4c","Type":"ContainerDied","Data":"0a58c35356491978d06c2ced6c413741ce63cbd451d40142ff73b46fd5a1f2e9"} Oct 30 22:18:43.851821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.851800 5607 generic.go:356] "Generic (PLEG): container finished" podID="f7c0067c-db3f-4662-8e4d-ee2f868271cf" containerID="f6fadd718f74b5a8f4e72fbe5ae6d27fc6a7476b97328b8d90d18dde1a4eac2a" exitCode=0 Oct 30 22:18:43.851930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.851871 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" event={"ID":"f7c0067c-db3f-4662-8e4d-ee2f868271cf","Type":"ContainerDied","Data":"f6fadd718f74b5a8f4e72fbe5ae6d27fc6a7476b97328b8d90d18dde1a4eac2a"} Oct 30 22:18:43.890618 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:43.890592 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:18:44.000651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.000626 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:18:44.001715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.001697 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume\") pod \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " Oct 30 22:18:44.001782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.001733 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2\" (UniqueName: \"kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2\") pod \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\" (UID: \"f7c0067c-db3f-4662-8e4d-ee2f868271cf\") " Oct 30 22:18:44.002120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.002087 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume" pod "f7c0067c-db3f-4662-8e4d-ee2f868271cf" (UID: "f7c0067c-db3f-4662-8e4d-ee2f868271cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:44.002599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.002578 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7c0067c-db3f-4662-8e4d-ee2f868271cf-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:44.003993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.003971 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2" pod "f7c0067c-db3f-4662-8e4d-ee2f868271cf" (UID: "f7c0067c-db3f-4662-8e4d-ee2f868271cf"). InnerVolumeSpecName "kube-api-access-bpbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:44.103680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.103642 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc\" (UniqueName: \"kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc\") pod \"c4655842-8ff9-49fb-af54-12c7972a8f4c\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " Oct 30 22:18:44.103909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.103896 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume\") pod \"c4655842-8ff9-49fb-af54-12c7972a8f4c\" (UID: \"c4655842-8ff9-49fb-af54-12c7972a8f4c\") " Oct 30 22:18:44.104258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.104223 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume" pod "c4655842-8ff9-49fb-af54-12c7972a8f4c" (UID: "c4655842-8ff9-49fb-af54-12c7972a8f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:44.104258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.104256 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbw2\" (UniqueName: \"kubernetes.io/projected/f7c0067c-db3f-4662-8e4d-ee2f868271cf-kube-api-access-bpbw2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:44.106049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.106027 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc" pod "c4655842-8ff9-49fb-af54-12c7972a8f4c" (UID: "c4655842-8ff9-49fb-af54-12c7972a8f4c"). InnerVolumeSpecName "kube-api-access-hkcqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:44.205901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.205871 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4655842-8ff9-49fb-af54-12c7972a8f4c-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:44.205901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.205901 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcqc\" (UniqueName: \"kubernetes.io/projected/c4655842-8ff9-49fb-af54-12c7972a8f4c-kube-api-access-hkcqc\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:44.860696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.860657 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" event={"ID":"c4655842-8ff9-49fb-af54-12c7972a8f4c","Type":"ContainerDied","Data":"8b3124586c1c4609518f440bb1d3ff75d9a772821a1a08dfbbabf72610ac5dcb"} Oct 30 22:18:44.861221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.860741 5607 scope.go:122] "RemoveContainer" containerID="0a58c35356491978d06c2ced6c413741ce63cbd451d40142ff73b46fd5a1f2e9" Oct 30 22:18:44.861221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.860668 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp" Oct 30 22:18:44.862485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.862466 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" Oct 30 22:18:44.874303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.874251 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt" event={"ID":"f7c0067c-db3f-4662-8e4d-ee2f868271cf","Type":"ContainerDied","Data":"7cbaa55765d6b28962769bd7c74f8db7773b34fa1db3d3646092fc40d31ca1d7"} Oct 30 22:18:44.874303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:44.874294 5607 scope.go:122] "RemoveContainer" containerID="f6fadd718f74b5a8f4e72fbe5ae6d27fc6a7476b97328b8d90d18dde1a4eac2a" Oct 30 22:18:45.032217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:45.032180 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt"] Oct 30 22:18:45.036651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:45.036624 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-647f75745-kbnpt"] Oct 30 22:18:45.149142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:45.149048 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp"] Oct 30 22:18:45.151715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:45.151697 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4576d9-5b9645df5f-24vkp"] Oct 30 22:18:45.731676 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:45.731641 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk"] Oct 30 22:18:45.731901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:45.731859 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" containerID="cri-o://19c85dd55a0a8efcb2960482ebd409e4b8e000573c97f08c27a36667611c8892" gracePeriod=30 Oct 30 22:18:46.870509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:46.870478 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4655842-8ff9-49fb-af54-12c7972a8f4c" path="/var/lib/kubelet/pods/c4655842-8ff9-49fb-af54-12c7972a8f4c/volumes" Oct 30 22:18:46.870905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:46.870887 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c0067c-db3f-4662-8e4d-ee2f868271cf" path="/var/lib/kubelet/pods/f7c0067c-db3f-4662-8e4d-ee2f868271cf/volumes" Oct 30 22:18:50.087683 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:50.087650 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:50.088204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:50.087743 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:50.907666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:50.907627 5607 generic.go:356] "Generic (PLEG): container finished" podID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerID="19c85dd55a0a8efcb2960482ebd409e4b8e000573c97f08c27a36667611c8892" exitCode=0 Oct 30 22:18:50.907840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:50.907694 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" event={"ID":"d77a4b8e-ffef-45e6-84f1-899fb85decad","Type":"ContainerDied","Data":"19c85dd55a0a8efcb2960482ebd409e4b8e000573c97f08c27a36667611c8892"} Oct 30 22:18:51.003126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.003064 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:18:51.106146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106118 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106154 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106232 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106309 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106362 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106391 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106455 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106520 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106607 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket\") pod \"d77a4b8e-ffef-45e6-84f1-899fb85decad\" (UID: \"d77a4b8e-ffef-45e6-84f1-899fb85decad\") " Oct 30 22:18:51.106860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106659 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:51.106860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106670 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:51.106860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106814 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:51.107189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.106975 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:51.107397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.107334 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:51.107764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.107742 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.107845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.107769 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.107845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.107783 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.107845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.107797 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77a4b8e-ffef-45e6-84f1-899fb85decad-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.107845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.107812 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.108670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.108647 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:51.108836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.108816 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "kube-api-access-hpfcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:51.108909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.108855 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:51.109058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.109043 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token" pod "d77a4b8e-ffef-45e6-84f1-899fb85decad" (UID: "d77a4b8e-ffef-45e6-84f1-899fb85decad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:51.209359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.209336 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.209359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.209360 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.209539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.209369 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77a4b8e-ffef-45e6-84f1-899fb85decad-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.209539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.209377 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfcn\" (UniqueName: \"kubernetes.io/projected/d77a4b8e-ffef-45e6-84f1-899fb85decad-kube-api-access-hpfcn\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:51.231512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.231489 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv"] Oct 30 22:18:51.232162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232143 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f7c0067c-db3f-4662-8e4d-ee2f868271cf" containerName="payload-processing" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232163 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c0067c-db3f-4662-8e4d-ee2f868271cf" containerName="payload-processing" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232206 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232215 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232233 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232240 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232252 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c4655842-8ff9-49fb-af54-12c7972a8f4c" containerName="payload-pre-processing" Oct 30 22:18:51.232264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232261 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4655842-8ff9-49fb-af54-12c7972a8f4c" containerName="payload-pre-processing" Oct 30 22:18:51.232600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232437 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c0067c-db3f-4662-8e4d-ee2f868271cf" containerName="payload-processing" Oct 30 22:18:51.232600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232457 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" Oct 30 22:18:51.232600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232466 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" containerName="istio-proxy" Oct 30 22:18:51.232600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.232474 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4655842-8ff9-49fb-af54-12c7972a8f4c" containerName="payload-pre-processing" Oct 30 22:18:51.236344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.236326 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.292075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.291984 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8b4b54-dockercfg-856xr\"" Oct 30 22:18:51.292075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.292009 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8b4b54\"" Oct 30 22:18:51.295356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.295332 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv"] Oct 30 22:18:51.310484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.310459 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.310594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.310526 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxqk\" (UniqueName: \"kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.411617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.411588 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.411771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.411661 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxqk\" (UniqueName: \"kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.412328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.412310 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.474525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.474500 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxqk\" (UniqueName: \"kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk\") pod \"payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.551856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.551791 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:51.915664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.915600 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" Oct 30 22:18:51.915808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.915599 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk" event={"ID":"d77a4b8e-ffef-45e6-84f1-899fb85decad","Type":"ContainerDied","Data":"383f9f52c4eac41ffd5f67c169cb2fec0f05c198c08936a8fbea16cf4e1355ed"} Oct 30 22:18:51.915808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.915738 5607 scope.go:122] "RemoveContainer" containerID="19c85dd55a0a8efcb2960482ebd409e4b8e000573c97f08c27a36667611c8892" Oct 30 22:18:51.928865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:51.928843 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv"] Oct 30 22:18:51.930987 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:18:51.930940 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c91be69_1499_4b54_ad45_2383cfc4ecba.slice/crio-95858a1b288d08beec9f385891287692806712bb3431ee327e914c34ac7ca0bc WatchSource:0}: Error finding container 95858a1b288d08beec9f385891287692806712bb3431ee327e914c34ac7ca0bc: Status 404 returned error can't find the container with id 95858a1b288d08beec9f385891287692806712bb3431ee327e914c34ac7ca0bc Oct 30 22:18:52.034421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.034391 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx"] Oct 30 22:18:52.035243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.035221 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" Oct 30 22:18:52.035243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.035241 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" Oct 30 22:18:52.035424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.035411 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af5df0b-401b-4bb9-bd6b-b6e91fe5d0e0" containerName="istio-proxy" Oct 30 22:18:52.038855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.038836 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.088216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.088192 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk"] Oct 30 22:18:52.092640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.092620 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-mz2vk"] Oct 30 22:18:52.095698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.095676 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx"] Oct 30 22:18:52.119874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.119847 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.120231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.120010 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2t4d\" (UniqueName: \"kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d\") pod \"payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.221289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.221257 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.221479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.221405 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2t4d\" (UniqueName: \"kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d\") pod \"payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.222023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.222006 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.284633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.284609 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2t4d\" (UniqueName: \"kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d\") pod \"payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.353213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.353186 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:52.728149 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:18:52.728118 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48a24c2_390c_468b_8a95_8298bf0d6378.slice/crio-3e11ab78381347c0ce976b4c9e5e9f1f2127a45c1a6def9ac280d9b1865f8547 WatchSource:0}: Error finding container 3e11ab78381347c0ce976b4c9e5e9f1f2127a45c1a6def9ac280d9b1865f8547: Status 404 returned error can't find the container with id 3e11ab78381347c0ce976b4c9e5e9f1f2127a45c1a6def9ac280d9b1865f8547 Oct 30 22:18:52.731339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.731318 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx"] Oct 30 22:18:52.863268 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:18:52.863235 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:18:52.870847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.870816 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77a4b8e-ffef-45e6-84f1-899fb85decad" path="/var/lib/kubelet/pods/d77a4b8e-ffef-45e6-84f1-899fb85decad/volumes" Oct 30 22:18:52.924205 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.924174 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" event={"ID":"6c91be69-1499-4b54-ad45-2383cfc4ecba","Type":"ContainerStarted","Data":"87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df"} Oct 30 22:18:52.924344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.924232 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" event={"ID":"6c91be69-1499-4b54-ad45-2383cfc4ecba","Type":"ContainerStarted","Data":"95858a1b288d08beec9f385891287692806712bb3431ee327e914c34ac7ca0bc"} Oct 30 22:18:52.924399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.924363 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:18:52.926902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.926878 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" event={"ID":"a48a24c2-390c-468b-8a95-8298bf0d6378","Type":"ContainerStarted","Data":"9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582"} Oct 30 22:18:52.927023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.926921 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" event={"ID":"a48a24c2-390c-468b-8a95-8298bf0d6378","Type":"ContainerStarted","Data":"3e11ab78381347c0ce976b4c9e5e9f1f2127a45c1a6def9ac280d9b1865f8547"} Oct 30 22:18:52.927070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:52.927029 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:18:53.044074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:53.043992 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" podStartSLOduration=2.043945788 podStartE2EDuration="2.043945788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:53.042450331 +0000 UTC m=+1912.443006453" watchObservedRunningTime="2026-10-30 22:18:53.043945788 +0000 UTC m=+1912.444501909" Oct 30 22:18:53.165858 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:18:53.165007 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" podStartSLOduration=2.164987854 podStartE2EDuration="2.164987854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:53.163805344 +0000 UTC m=+1912.564361465" watchObservedRunningTime="2026-10-30 22:18:53.164987854 +0000 UTC m=+1912.565543976" Oct 30 22:19:03.127197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:03.127148 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:19:03.127620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:03.127235 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:03.935974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:03.935926 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:19:03.936162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:03.936058 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:19:05.015036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.015002 5607 generic.go:356] "Generic (PLEG): container finished" podID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerID="ea4c355f3383c81122f9d1f51d6fd01ef7b690de715fd8a10861f9471dbf829a" exitCode=0 Oct 30 22:19:05.015526 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.015083 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" event={"ID":"d42d7934-e97a-45fd-a4b1-58420951e26e","Type":"ContainerDied","Data":"ea4c355f3383c81122f9d1f51d6fd01ef7b690de715fd8a10861f9471dbf829a"} Oct 30 22:19:05.015526 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.015501 5607 scope.go:122] "RemoveContainer" containerID="ea4c355f3383c81122f9d1f51d6fd01ef7b690de715fd8a10861f9471dbf829a" Oct 30 22:19:05.016945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.016912 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:05.017054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.017007 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:05.017054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.017043 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:05.848881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.848845 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs"] Oct 30 22:19:05.851810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.851794 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:05.905476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.905445 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0e90fe-serving-cert\"" Oct 30 22:19:05.961102 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:05.961067 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs"] Oct 30 22:19:06.023173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.023141 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" event={"ID":"d42d7934-e97a-45fd-a4b1-58420951e26e","Type":"ContainerStarted","Data":"69455949d8d8a29c3d573a950d9732740bc18b71f9f5d0979ea919e26e917d75"} Oct 30 22:19:06.029074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.029039 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.029211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.029194 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ddd\" (UniqueName: \"kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.029311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.029297 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.029367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.029355 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.131233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.131140 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ddd\" (UniqueName: \"kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.131435 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.131260 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.131435 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.131379 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.131577 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.131445 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.132679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.132551 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.135411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.135386 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.135530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.135429 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.194139 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.194107 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ddd\" (UniqueName: \"kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd\") pod \"maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.464596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.464557 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:06.559186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.559155 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:19:06.559333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.559282 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:19:06.559405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.559327 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 22:19:06.865442 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:19:06.865407 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02787274_f04a_491f_bc8f_e340311f3ac7.slice/crio-9498bc58cc9f5f53d92342c4ec82e5d3c65c4c35dd1a49a14ff2cad627f7e111 WatchSource:0}: Error finding container 9498bc58cc9f5f53d92342c4ec82e5d3c65c4c35dd1a49a14ff2cad627f7e111: Status 404 returned error can't find the container with id 9498bc58cc9f5f53d92342c4ec82e5d3c65c4c35dd1a49a14ff2cad627f7e111 Oct 30 22:19:06.871748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:06.871725 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs"] Oct 30 22:19:07.030271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:07.030229 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" event={"ID":"02787274-f04a-491f-bc8f-e340311f3ac7","Type":"ContainerStarted","Data":"9498bc58cc9f5f53d92342c4ec82e5d3c65c4c35dd1a49a14ff2cad627f7e111"} Oct 30 22:19:07.559027 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:07.558991 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:19:07.559202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:07.559061 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 22:19:08.038556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:08.038523 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" event={"ID":"02787274-f04a-491f-bc8f-e340311f3ac7","Type":"ContainerStarted","Data":"cb6485f6f5708da67f75288c3f6b8e88cfdfe52df118888d5f3ff7455d71ead9"} Oct 30 22:19:08.039026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:08.038590 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:08.159305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:08.159206 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" podStartSLOduration=2.293776252 podStartE2EDuration="3.159192421s" totalImagesPullingTime="865.416169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:08.157866633 +0000 UTC m=+1927.558422754" watchObservedRunningTime="2026-10-30 22:19:08.159192421 +0000 UTC m=+1927.559748543" Oct 30 22:19:08.559251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:08.559214 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:19:08.559478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:08.559273 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 22:19:09.559343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:09.559307 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:19:09.559854 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:09.559359 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 22:19:10.558932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:10.558895 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:19:10.559149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:10.558970 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 22:19:11.558789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:11.558748 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 22:19:11.559287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:11.558808 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 22:19:12.562768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:12.562734 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:19:12.563198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:12.563161 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:19:12.563794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:12.563777 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:19:14.050691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:14.050659 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:19:19.525976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.523227 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59"] Oct 30 22:19:19.526800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.526746 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" podUID="4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" containerName="maas-api" containerID="cri-o://ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72" gracePeriod=30 Oct 30 22:19:19.612027 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:19.611993 5607 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd88396_ecd7_4f9e_9c36_cd2f3ad5deb8.slice/crio-ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd88396_ecd7_4f9e_9c36_cd2f3ad5deb8.slice/crio-conmon-ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:19.885294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.885267 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:19:19.895659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.895629 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr"] Oct 30 22:19:19.895880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.895854 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" podUID="126007cb-90a8-424f-a8a8-412ab2af1b6f" containerName="payload-processing" containerID="cri-o://47a7f46a66f54491e6c734623ecd6bfa73e436412af0d44fbb291fee0b6b28f9" gracePeriod=30 Oct 30 22:19:19.939549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.939507 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566\" (UniqueName: \"kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566\") pod \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " Oct 30 22:19:19.939725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.939682 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls\") pod \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " Oct 30 22:19:19.939725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.939706 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs\") pod \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " Oct 30 22:19:19.939869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.939784 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca\") pod \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\" (UID: \"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8\") " Oct 30 22:19:19.941041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.941014 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca" pod "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" (UID: "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:19.941938 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.941910 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566" pod "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" (UID: "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8"). InnerVolumeSpecName "kube-api-access-48566". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:19.942554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.942531 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls" pod "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" (UID: "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:19.943067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:19.943040 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs" pod "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" (UID: "4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:20.041160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.041099 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48566\" (UniqueName: \"kubernetes.io/projected/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-kube-api-access-48566\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.041160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.041135 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.041160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.041149 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.041160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.041162 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.119564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.119535 5607 generic.go:356] "Generic (PLEG): container finished" podID="126007cb-90a8-424f-a8a8-412ab2af1b6f" containerID="47a7f46a66f54491e6c734623ecd6bfa73e436412af0d44fbb291fee0b6b28f9" exitCode=0 Oct 30 22:19:20.119737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.119616 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" event={"ID":"126007cb-90a8-424f-a8a8-412ab2af1b6f","Type":"ContainerDied","Data":"47a7f46a66f54491e6c734623ecd6bfa73e436412af0d44fbb291fee0b6b28f9"} Oct 30 22:19:20.121237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.121203 5607 generic.go:356] "Generic (PLEG): container finished" podID="4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" containerID="ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72" exitCode=0 Oct 30 22:19:20.121379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.121361 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" event={"ID":"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8","Type":"ContainerDied","Data":"ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72"} Oct 30 22:19:20.121442 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.121407 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" event={"ID":"4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8","Type":"ContainerDied","Data":"cb8b5f2f7a2998275d510c30ba9651809b5842921410bef78761b7e0b6b97ea2"} Oct 30 22:19:20.121495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.121453 5607 scope.go:122] "RemoveContainer" containerID="ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72" Oct 30 22:19:20.121610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.121598 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59" Oct 30 22:19:20.135512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.135490 5607 scope.go:122] "RemoveContainer" containerID="ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72" Oct 30 22:19:20.137170 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:20.137145 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72\": container with ID starting with ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72 not found: ID does not exist" containerID="ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72" Oct 30 22:19:20.137696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.137669 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72"} err="failed to get container status \"ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72\": rpc error: code = NotFound desc = could not find container \"ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72\": container with ID starting with ee01b9addc1f06694c681fccf89e19ce929825865965d876282a49ba8df1eb72 not found: ID does not exist" Oct 30 22:19:20.213330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.213306 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:19:20.243556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.243528 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh\" (UniqueName: \"kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh\") pod \"126007cb-90a8-424f-a8a8-412ab2af1b6f\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " Oct 30 22:19:20.243696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.243607 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume\") pod \"126007cb-90a8-424f-a8a8-412ab2af1b6f\" (UID: \"126007cb-90a8-424f-a8a8-412ab2af1b6f\") " Oct 30 22:19:20.243965 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.243930 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume" pod "126007cb-90a8-424f-a8a8-412ab2af1b6f" (UID: "126007cb-90a8-424f-a8a8-412ab2af1b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:20.244325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.244303 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126007cb-90a8-424f-a8a8-412ab2af1b6f-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.245791 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.245769 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh" pod "126007cb-90a8-424f-a8a8-412ab2af1b6f" (UID: "126007cb-90a8-424f-a8a8-412ab2af1b6f"). InnerVolumeSpecName "kube-api-access-x28qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:20.255592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.255570 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59"] Oct 30 22:19:20.261352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.261333 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad7da0-6b5b5457f7-64z59"] Oct 30 22:19:20.345700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.345676 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28qh\" (UniqueName: \"kubernetes.io/projected/126007cb-90a8-424f-a8a8-412ab2af1b6f-kube-api-access-x28qh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:20.871075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:20.870991 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" path="/var/lib/kubelet/pods/4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8/volumes" Oct 30 22:19:21.131399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.131300 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" Oct 30 22:19:21.131399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.131311 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr" event={"ID":"126007cb-90a8-424f-a8a8-412ab2af1b6f","Type":"ContainerDied","Data":"59f54ad72e430dbb900aa1bbbbb99f5f62d142d7933e0b03cdf6b14477935f7b"} Oct 30 22:19:21.131623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.131412 5607 scope.go:122] "RemoveContainer" containerID="47a7f46a66f54491e6c734623ecd6bfa73e436412af0d44fbb291fee0b6b28f9" Oct 30 22:19:21.136742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.136721 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5"] Oct 30 22:19:21.136937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.136893 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" podUID="5660253c-f8a7-408e-8d85-b85e624eb4b9" containerName="payload-processing" containerID="cri-o://254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25" gracePeriod=30 Oct 30 22:19:21.426832 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.426803 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr"] Oct 30 22:19:21.430395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.430374 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ad7da0-6c9cf96876-5zclr"] Oct 30 22:19:21.442055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.442036 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:19:21.572253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.571971 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48\" (UniqueName: \"kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48\") pod \"5660253c-f8a7-408e-8d85-b85e624eb4b9\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " Oct 30 22:19:21.572253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.572086 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume\") pod \"5660253c-f8a7-408e-8d85-b85e624eb4b9\" (UID: \"5660253c-f8a7-408e-8d85-b85e624eb4b9\") " Oct 30 22:19:21.572572 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.572548 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume" pod "5660253c-f8a7-408e-8d85-b85e624eb4b9" (UID: "5660253c-f8a7-408e-8d85-b85e624eb4b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:21.574609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.574572 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48" pod "5660253c-f8a7-408e-8d85-b85e624eb4b9" (UID: "5660253c-f8a7-408e-8d85-b85e624eb4b9"). InnerVolumeSpecName "kube-api-access-dkl48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:21.673562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.673485 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkl48\" (UniqueName: \"kubernetes.io/projected/5660253c-f8a7-408e-8d85-b85e624eb4b9-kube-api-access-dkl48\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:21.673562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:21.673515 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5660253c-f8a7-408e-8d85-b85e624eb4b9-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:22.057007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.056976 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z"] Oct 30 22:19:22.141780 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.141748 5607 generic.go:356] "Generic (PLEG): container finished" podID="5660253c-f8a7-408e-8d85-b85e624eb4b9" containerID="254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25" exitCode=0 Oct 30 22:19:22.142007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.141813 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" Oct 30 22:19:22.142007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.141847 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" event={"ID":"5660253c-f8a7-408e-8d85-b85e624eb4b9","Type":"ContainerDied","Data":"254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25"} Oct 30 22:19:22.142007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.141917 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5" event={"ID":"5660253c-f8a7-408e-8d85-b85e624eb4b9","Type":"ContainerDied","Data":"b495e9e056f408c594407e928cac2939734d5891cbeab2bc675d5c0565872872"} Oct 30 22:19:22.142007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.141947 5607 scope.go:122] "RemoveContainer" containerID="254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25" Oct 30 22:19:22.153135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.153118 5607 scope.go:122] "RemoveContainer" containerID="254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25" Oct 30 22:19:22.153431 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:22.153411 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25\": container with ID starting with 254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25 not found: ID does not exist" containerID="254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25" Oct 30 22:19:22.153479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.153439 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25"} err="failed to get container status \"254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25\": rpc error: code = NotFound desc = could not find container \"254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25\": container with ID starting with 254d0b1a8f4604dbaa909926591f3d58ab7af44696c47f57d495c265232ddb25 not found: ID does not exist" Oct 30 22:19:22.172001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.170833 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8"] Oct 30 22:19:22.307946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.307874 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5"] Oct 30 22:19:22.311449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.311424 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-6f79487fbb-s2cq5"] Oct 30 22:19:22.863909 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:22.863876 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:19:22.872340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.872312 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126007cb-90a8-424f-a8a8-412ab2af1b6f" path="/var/lib/kubelet/pods/126007cb-90a8-424f-a8a8-412ab2af1b6f/volumes" Oct 30 22:19:22.872730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.872716 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5660253c-f8a7-408e-8d85-b85e624eb4b9" path="/var/lib/kubelet/pods/5660253c-f8a7-408e-8d85-b85e624eb4b9/volumes" Oct 30 22:19:22.928221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928176 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw"] Oct 30 22:19:22.928812 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928798 5607 cpu_manager.go:414] "containerMap: removing container" podUID="5660253c-f8a7-408e-8d85-b85e624eb4b9" containerName="payload-processing" Oct 30 22:19:22.928864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928815 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5660253c-f8a7-408e-8d85-b85e624eb4b9" containerName="payload-processing" Oct 30 22:19:22.928864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928830 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" containerName="maas-api" Oct 30 22:19:22.928864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928835 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" containerName="maas-api" Oct 30 22:19:22.928864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928847 5607 cpu_manager.go:414] "containerMap: removing container" podUID="126007cb-90a8-424f-a8a8-412ab2af1b6f" containerName="payload-processing" Oct 30 22:19:22.928864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928853 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126007cb-90a8-424f-a8a8-412ab2af1b6f" containerName="payload-processing" Oct 30 22:19:22.929086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928975 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="5660253c-f8a7-408e-8d85-b85e624eb4b9" containerName="payload-processing" Oct 30 22:19:22.929086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928984 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd88396-ecd7-4f9e-9c36-cd2f3ad5deb8" containerName="maas-api" Oct 30 22:19:22.929086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.928990 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="126007cb-90a8-424f-a8a8-412ab2af1b6f" containerName="payload-processing" Oct 30 22:19:22.931830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.931809 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.987881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.987851 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7350bc-openshift-default-dockercfg-j5256\"" Oct 30 22:19:22.991603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991577 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvfg\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.991742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991618 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.991742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991696 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.991742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991722 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.991888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991765 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.991933 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991903 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.992019 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.991943 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.992071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.992057 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.992110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.992086 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:22.998381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:22.998351 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw"] Oct 30 22:19:23.094243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094206 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094302 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094331 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094461 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvfg\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094495 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094525 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094565 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.094761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094690 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.095185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094776 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.095185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094799 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.095185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094824 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.095185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094877 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.095185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.094972 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.095388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.095300 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.097356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.097335 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.097911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.097882 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.221118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.212238 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvfg\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.237976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.235618 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token\") pod \"e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.253011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.251718 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:23.695898 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:19:23.692080 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f264482_3845_4221_bbf1_95a1b3b386ae.slice/crio-914bbac0c89fca1b66b4a717de3a6d87b235c0e456cecbe8f12252068b730d98 WatchSource:0}: Error finding container 914bbac0c89fca1b66b4a717de3a6d87b235c0e456cecbe8f12252068b730d98: Status 404 returned error can't find the container with id 914bbac0c89fca1b66b4a717de3a6d87b235c0e456cecbe8f12252068b730d98 Oct 30 22:19:23.699446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.697363 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:23.699446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.697461 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:23.699446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.697501 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:23.703043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:23.700757 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw"] Oct 30 22:19:24.161222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:24.161133 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" event={"ID":"9f264482-3845-4221-bbf1-95a1b3b386ae","Type":"ContainerStarted","Data":"de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078"} Oct 30 22:19:24.161666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:24.161245 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" event={"ID":"9f264482-3845-4221-bbf1-95a1b3b386ae","Type":"ContainerStarted","Data":"914bbac0c89fca1b66b4a717de3a6d87b235c0e456cecbe8f12252068b730d98"} Oct 30 22:19:24.252009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:24.251978 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:24.257485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:24.257461 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:24.284347 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:24.284297 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podStartSLOduration=2.2842851619999998 podStartE2EDuration="2.284285162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:24.283448525 +0000 UTC m=+1943.684004646" watchObservedRunningTime="2026-10-30 22:19:24.284285162 +0000 UTC m=+1943.684841283" Oct 30 22:19:25.167757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:25.167717 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:25.168613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:25.168593 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:19:27.058259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.058217 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" podUID="fe218379-79d6-4eb1-81f9-989b79f45d4e" containerName="payload-processing" containerID="cri-o://952abd95b0805151876940fee7f7b68068b6825bef9a58cedb4e2f24a305a3cd" gracePeriod=25 Oct 30 22:19:27.171532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.171497 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" podUID="909229f2-f9dd-4759-84f7-5f62f4e61676" containerName="payload-pre-processing" containerID="cri-o://1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3" gracePeriod=25 Oct 30 22:19:27.182557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.182528 5607 generic.go:356] "Generic (PLEG): container finished" podID="fe218379-79d6-4eb1-81f9-989b79f45d4e" containerID="952abd95b0805151876940fee7f7b68068b6825bef9a58cedb4e2f24a305a3cd" exitCode=0 Oct 30 22:19:27.182677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.182606 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" event={"ID":"fe218379-79d6-4eb1-81f9-989b79f45d4e","Type":"ContainerDied","Data":"952abd95b0805151876940fee7f7b68068b6825bef9a58cedb4e2f24a305a3cd"} Oct 30 22:19:27.469904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.469667 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:19:27.600770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.600686 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s\" (UniqueName: \"kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s\") pod \"fe218379-79d6-4eb1-81f9-989b79f45d4e\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " Oct 30 22:19:27.601105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.601081 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume\") pod \"fe218379-79d6-4eb1-81f9-989b79f45d4e\" (UID: \"fe218379-79d6-4eb1-81f9-989b79f45d4e\") " Oct 30 22:19:27.605210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.605182 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume" pod "fe218379-79d6-4eb1-81f9-989b79f45d4e" (UID: "fe218379-79d6-4eb1-81f9-989b79f45d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:27.671411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.671096 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:19:27.674561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.674526 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s" pod "fe218379-79d6-4eb1-81f9-989b79f45d4e" (UID: "fe218379-79d6-4eb1-81f9-989b79f45d4e"). InnerVolumeSpecName "kube-api-access-p7z9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:27.720985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.720135 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7z9s\" (UniqueName: \"kubernetes.io/projected/fe218379-79d6-4eb1-81f9-989b79f45d4e-kube-api-access-p7z9s\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.720985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.720169 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe218379-79d6-4eb1-81f9-989b79f45d4e-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.825979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.823594 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume\") pod \"909229f2-f9dd-4759-84f7-5f62f4e61676\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " Oct 30 22:19:27.825979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.823710 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv\" (UniqueName: \"kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv\") pod \"909229f2-f9dd-4759-84f7-5f62f4e61676\" (UID: \"909229f2-f9dd-4759-84f7-5f62f4e61676\") " Oct 30 22:19:27.825979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.825439 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume" pod "909229f2-f9dd-4759-84f7-5f62f4e61676" (UID: "909229f2-f9dd-4759-84f7-5f62f4e61676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:27.829862 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.829829 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv" pod "909229f2-f9dd-4759-84f7-5f62f4e61676" (UID: "909229f2-f9dd-4759-84f7-5f62f4e61676"). InnerVolumeSpecName "kube-api-access-khgnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:27.927156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.927075 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909229f2-f9dd-4759-84f7-5f62f4e61676-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.927156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:27.927100 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgnv\" (UniqueName: \"kubernetes.io/projected/909229f2-f9dd-4759-84f7-5f62f4e61676-kube-api-access-khgnv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:28.190895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.190862 5607 generic.go:356] "Generic (PLEG): container finished" podID="909229f2-f9dd-4759-84f7-5f62f4e61676" containerID="1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3" exitCode=0 Oct 30 22:19:28.191373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.190909 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" Oct 30 22:19:28.191373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.190947 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" event={"ID":"909229f2-f9dd-4759-84f7-5f62f4e61676","Type":"ContainerDied","Data":"1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3"} Oct 30 22:19:28.191373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.191031 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8" event={"ID":"909229f2-f9dd-4759-84f7-5f62f4e61676","Type":"ContainerDied","Data":"0dbbe5d117ee72c2c4d12a6f74f4f671777fff6138eaf9cd002d86b0b75bcf6f"} Oct 30 22:19:28.191373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.191064 5607 scope.go:122] "RemoveContainer" containerID="1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3" Oct 30 22:19:28.192613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.192598 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" Oct 30 22:19:28.192714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.192612 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z" event={"ID":"fe218379-79d6-4eb1-81f9-989b79f45d4e","Type":"ContainerDied","Data":"3ad0330a2d78b6a561157264f97f71915ffc9cff0bbc00f5c6953061b17543e4"} Oct 30 22:19:28.228193 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.228169 5607 scope.go:122] "RemoveContainer" containerID="1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3" Oct 30 22:19:28.234811 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:28.234714 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3\": container with ID starting with 1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3 not found: ID does not exist" containerID="1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3" Oct 30 22:19:28.234811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.234773 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3"} err="failed to get container status \"1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3\": rpc error: code = NotFound desc = could not find container \"1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3\": container with ID starting with 1bdc28d6a1fe33a05713714dee0b295d469e16b851319328c8889e92cbe467a3 not found: ID does not exist" Oct 30 22:19:28.235013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.234831 5607 scope.go:122] "RemoveContainer" containerID="952abd95b0805151876940fee7f7b68068b6825bef9a58cedb4e2f24a305a3cd" Oct 30 22:19:28.383502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.383463 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z"] Oct 30 22:19:28.386376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.386349 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8daffd-684b984799-pq72z"] Oct 30 22:19:28.526108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.526014 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8"] Oct 30 22:19:28.528487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.528460 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-6486bf56c-f6pn8"] Oct 30 22:19:28.871005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.870919 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909229f2-f9dd-4759-84f7-5f62f4e61676" path="/var/lib/kubelet/pods/909229f2-f9dd-4759-84f7-5f62f4e61676/volumes" Oct 30 22:19:28.871370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:28.871357 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe218379-79d6-4eb1-81f9-989b79f45d4e" path="/var/lib/kubelet/pods/fe218379-79d6-4eb1-81f9-989b79f45d4e/volumes" Oct 30 22:19:29.023977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.023295 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t"] Oct 30 22:19:29.029975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.024298 5607 cpu_manager.go:414] "containerMap: removing container" podUID="909229f2-f9dd-4759-84f7-5f62f4e61676" containerName="payload-pre-processing" Oct 30 22:19:29.029975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.024323 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909229f2-f9dd-4759-84f7-5f62f4e61676" containerName="payload-pre-processing" Oct 30 22:19:29.029975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.024371 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fe218379-79d6-4eb1-81f9-989b79f45d4e" containerName="payload-processing" Oct 30 22:19:29.029975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.024379 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe218379-79d6-4eb1-81f9-989b79f45d4e" containerName="payload-processing" Oct 30 22:19:29.029975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.024509 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe218379-79d6-4eb1-81f9-989b79f45d4e" containerName="payload-processing" Oct 30 22:19:29.029975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.024532 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="909229f2-f9dd-4759-84f7-5f62f4e61676" containerName="payload-pre-processing" Oct 30 22:19:29.035564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.035538 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.073124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.073103 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t"] Oct 30 22:19:29.089716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.089692 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4576d9-dockercfg-ht95n\"" Oct 30 22:19:29.089846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.089790 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4576d9\"" Oct 30 22:19:29.145202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.145128 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5d7\" (UniqueName: \"kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7\") pod \"payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.145202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.145174 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.246485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.246448 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5d7\" (UniqueName: \"kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7\") pod \"payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.246970 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.246514 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.247717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.247527 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.322387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.322354 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5d7\" (UniqueName: \"kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7\") pod \"payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.350232 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.350203 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:29.825335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:29.825309 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t"] Oct 30 22:19:29.827360 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:19:29.827334 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc91881_ca67_4d2a_94ed_a67550fe2f5d.slice/crio-c4838d343afd5e444c94c5f52dd8c43eee6c85b5414c197c4482cac8a347d7c3 WatchSource:0}: Error finding container c4838d343afd5e444c94c5f52dd8c43eee6c85b5414c197c4482cac8a347d7c3: Status 404 returned error can't find the container with id c4838d343afd5e444c94c5f52dd8c43eee6c85b5414c197c4482cac8a347d7c3 Oct 30 22:19:30.234847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:30.234808 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" event={"ID":"bbc91881-ca67-4d2a-94ed-a67550fe2f5d","Type":"ContainerStarted","Data":"9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35"} Oct 30 22:19:30.235088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:30.234871 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" event={"ID":"bbc91881-ca67-4d2a-94ed-a67550fe2f5d","Type":"ContainerStarted","Data":"c4838d343afd5e444c94c5f52dd8c43eee6c85b5414c197c4482cac8a347d7c3"} Oct 30 22:19:30.235162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:30.235095 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:30.355174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:30.355125 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" podStartSLOduration=2.355106727 podStartE2EDuration="2.355106727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:30.353529364 +0000 UTC m=+1949.754085484" watchObservedRunningTime="2026-10-30 22:19:30.355106727 +0000 UTC m=+1949.755662848" Oct 30 22:19:31.479219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.479188 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds"] Oct 30 22:19:31.480326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.480286 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" containerID="cri-o://eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5" gracePeriod=30 Oct 30 22:19:31.685157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.685115 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl"] Oct 30 22:19:31.693739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.693708 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:31.727826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.727678 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf8ms\" (UniqueName: \"kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:31.727826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.727739 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:31.747999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.747945 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl"] Oct 30 22:19:31.767482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.767463 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7350bc\"" Oct 30 22:19:31.767761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.767735 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7350bc-dockercfg-mxsmz\"" Oct 30 22:19:31.836705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.836653 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf8ms\" (UniqueName: \"kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:31.836878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.836732 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:31.837714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.837652 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:31.923530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:31.923459 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf8ms\" (UniqueName: \"kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms\") pod \"payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:32.039888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:32.039808 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:32.426037 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:19:32.426005 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ce887c_6d7e_4ea3_85b2_9e8c245a1e41.slice/crio-a41738110f23fc81e8bbbea7826692de1eb050541a6222ed9eda0de3b7cfb974 WatchSource:0}: Error finding container a41738110f23fc81e8bbbea7826692de1eb050541a6222ed9eda0de3b7cfb974: Status 404 returned error can't find the container with id a41738110f23fc81e8bbbea7826692de1eb050541a6222ed9eda0de3b7cfb974 Oct 30 22:19:32.428722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:32.428700 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl"] Oct 30 22:19:33.276982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:33.276935 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" event={"ID":"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41","Type":"ContainerStarted","Data":"5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986"} Oct 30 22:19:33.277491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:33.277007 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" event={"ID":"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41","Type":"ContainerStarted","Data":"a41738110f23fc81e8bbbea7826692de1eb050541a6222ed9eda0de3b7cfb974"} Oct 30 22:19:33.277491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:33.277152 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:33.400507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:33.400453 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" podStartSLOduration=2.4004359920000002 podStartE2EDuration="2.400435992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:33.396966226 +0000 UTC m=+1952.797522340" watchObservedRunningTime="2026-10-30 22:19:33.400435992 +0000 UTC m=+1952.800992110" Oct 30 22:19:35.898831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:35.898360 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": read tcp 10.133.0.2:57492->10.133.0.159:15021: read: connection reset by peer" start-of-body= Oct 30 22:19:35.898831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:35.898421 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": read tcp 10.133.0.2:57492->10.133.0.159:15021: read: connection reset by peer" Oct 30 22:19:36.297646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.297615 5607 generic.go:356] "Generic (PLEG): container finished" podID="2aa5eb97-61b8-49de-8499-491918f935a7" containerID="6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656" exitCode=0 Oct 30 22:19:36.297830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.297693 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" event={"ID":"2aa5eb97-61b8-49de-8499-491918f935a7","Type":"ContainerDied","Data":"6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656"} Oct 30 22:19:36.298103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.298085 5607 scope.go:122] "RemoveContainer" containerID="6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656" Oct 30 22:19:36.299385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.299357 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:36.299443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.299428 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:36.299483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.299460 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:36.762045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.762021 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:19:36.792001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.791941 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792042 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792074 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792108 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792133 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792162 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792196 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792358 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792407 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:36.792536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792413 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:36.792536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792498 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo\") pod \"551af49d-9c92-48ed-9a8f-831c634c31bd\" (UID: \"551af49d-9c92-48ed-9a8f-831c634c31bd\") " Oct 30 22:19:36.792705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792578 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:36.792705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792581 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:36.792972 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.792897 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:36.793137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.793056 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551af49d-9c92-48ed-9a8f-831c634c31bd-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.793137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.793075 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.793137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.793089 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.793137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.793104 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.793370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.793237 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.794589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.794567 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:36.794686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.794665 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:36.794942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.794925 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "kube-api-access-7tgc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:36.795054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.795030 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo" pod "551af49d-9c92-48ed-9a8f-831c634c31bd" (UID: "551af49d-9c92-48ed-9a8f-831c634c31bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:36.894148 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.894121 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.894148 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.894150 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.894290 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.894164 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tgc8\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-kube-api-access-7tgc8\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:36.894290 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:36.894177 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551af49d-9c92-48ed-9a8f-831c634c31bd-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:37.305438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.305406 5607 generic.go:356] "Generic (PLEG): container finished" podID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerID="eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5" exitCode=0 Oct 30 22:19:37.305838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.305465 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" Oct 30 22:19:37.305838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.305488 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" event={"ID":"551af49d-9c92-48ed-9a8f-831c634c31bd","Type":"ContainerDied","Data":"eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5"} Oct 30 22:19:37.305838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.305544 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds" event={"ID":"551af49d-9c92-48ed-9a8f-831c634c31bd","Type":"ContainerDied","Data":"6e60822e987a90a48352d54217b52bd499ae3db3f318770d0509e9b187491fb0"} Oct 30 22:19:37.305838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.305566 5607 scope.go:122] "RemoveContainer" containerID="eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5" Oct 30 22:19:37.308227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.308207 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" event={"ID":"2aa5eb97-61b8-49de-8499-491918f935a7","Type":"ContainerStarted","Data":"9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb"} Oct 30 22:19:37.316193 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.316178 5607 scope.go:122] "RemoveContainer" containerID="f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c" Oct 30 22:19:37.328021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.328003 5607 scope.go:122] "RemoveContainer" containerID="eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5" Oct 30 22:19:37.328376 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:37.328357 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5\": container with ID starting with eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5 not found: ID does not exist" containerID="eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5" Oct 30 22:19:37.328437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.328388 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5"} err="failed to get container status \"eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5\": rpc error: code = NotFound desc = could not find container \"eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5\": container with ID starting with eb49c1bb933dc8c222327c15b0b7c31e8ebc3875126b055401d573c8c831b1a5 not found: ID does not exist" Oct 30 22:19:37.328437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.328411 5607 scope.go:122] "RemoveContainer" containerID="f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c" Oct 30 22:19:37.328669 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:37.328654 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c\": container with ID starting with f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c not found: ID does not exist" containerID="f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c" Oct 30 22:19:37.328719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.328675 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c"} err="failed to get container status \"f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c\": rpc error: code = NotFound desc = could not find container \"f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c\": container with ID starting with f90650d007d4c5fdf2379ad8f4959af6d8328aa09765c0c44343c7e691e6d39c not found: ID does not exist" Oct 30 22:19:37.604210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.603431 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds"] Oct 30 22:19:37.608587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:37.608549 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-04d404-openshift-default-5f687b48cf-7vxds"] Oct 30 22:19:38.034934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:38.034907 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:19:38.035281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:38.035249 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:19:38.035394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:38.035301 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 22:19:38.870468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:38.870439 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" path="/var/lib/kubelet/pods/551af49d-9c92-48ed-9a8f-831c634c31bd/volumes" Oct 30 22:19:39.034411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:39.034379 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:19:39.034580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:39.034430 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 22:19:40.034220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:40.034184 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:19:40.034705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:40.034234 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 22:19:41.034421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:41.034379 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:19:41.034839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:41.034428 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 22:19:41.262373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:41.262346 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:19:42.034467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:42.034433 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:19:42.034860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:42.034486 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 22:19:43.034649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:43.034604 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:19:43.035130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:43.034674 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 22:19:44.038458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:44.038429 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:19:44.038870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:44.038853 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:19:44.039476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:44.039460 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:19:44.288166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:19:44.288141 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:19:52.864868 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:19:52.864832 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:20:14.998075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998001 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l"] Oct 30 22:20:14.998624 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998606 5607 cpu_manager.go:414] "containerMap: removing container" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" Oct 30 22:20:14.998671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998626 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" Oct 30 22:20:14.998671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998645 5607 cpu_manager.go:414] "containerMap: removing container" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" Oct 30 22:20:14.998671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998650 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" Oct 30 22:20:14.998776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998763 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" Oct 30 22:20:14.998811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:14.998785 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="551af49d-9c92-48ed-9a8f-831c634c31bd" containerName="istio-proxy" Oct 30 22:20:15.002172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.002156 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.057261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.057236 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8daffd\"" Oct 30 22:20:15.080083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.080051 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzm2\" (UniqueName: \"kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.080294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.080270 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.108222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.108197 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8daffd-dockercfg-z8sv8\"" Oct 30 22:20:15.111106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.111085 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l"] Oct 30 22:20:15.180923 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.180896 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.181069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.181000 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzm2\" (UniqueName: \"kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.181590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.181575 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.243751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.243728 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzm2\" (UniqueName: \"kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2\") pod \"payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.314772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.314699 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:15.684026 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:15.683990 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfdb567_f0f2_4e68_985a_fc5e9603bbfa.slice/crio-1d407815dd050e9ac710541811e7fd0c5a0824918eef39cd7ee327ef7d2ef3fd WatchSource:0}: Error finding container 1d407815dd050e9ac710541811e7fd0c5a0824918eef39cd7ee327ef7d2ef3fd: Status 404 returned error can't find the container with id 1d407815dd050e9ac710541811e7fd0c5a0824918eef39cd7ee327ef7d2ef3fd Oct 30 22:20:15.686909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:15.686805 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l"] Oct 30 22:20:16.328783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.328742 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t"] Oct 30 22:20:16.329335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.329008 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" podUID="bbc91881-ca67-4d2a-94ed-a67550fe2f5d" containerName="payload-processing" containerID="cri-o://9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35" gracePeriod=30 Oct 30 22:20:16.614797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.614771 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:20:16.691713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.691673 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" event={"ID":"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa","Type":"ContainerStarted","Data":"3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29"} Oct 30 22:20:16.691864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.691747 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:16.691864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.691775 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" event={"ID":"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa","Type":"ContainerStarted","Data":"1d407815dd050e9ac710541811e7fd0c5a0824918eef39cd7ee327ef7d2ef3fd"} Oct 30 22:20:16.693151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.693130 5607 generic.go:356] "Generic (PLEG): container finished" podID="bbc91881-ca67-4d2a-94ed-a67550fe2f5d" containerID="9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35" exitCode=0 Oct 30 22:20:16.693266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.693164 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" Oct 30 22:20:16.693266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.693211 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" event={"ID":"bbc91881-ca67-4d2a-94ed-a67550fe2f5d","Type":"ContainerDied","Data":"9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35"} Oct 30 22:20:16.693338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.693272 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t" event={"ID":"bbc91881-ca67-4d2a-94ed-a67550fe2f5d","Type":"ContainerDied","Data":"c4838d343afd5e444c94c5f52dd8c43eee6c85b5414c197c4482cac8a347d7c3"} Oct 30 22:20:16.693338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.693305 5607 scope.go:122] "RemoveContainer" containerID="9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35" Oct 30 22:20:16.704341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.704321 5607 scope.go:122] "RemoveContainer" containerID="9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35" Oct 30 22:20:16.704611 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:16.704593 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35\": container with ID starting with 9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35 not found: ID does not exist" containerID="9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35" Oct 30 22:20:16.704670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.704617 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35"} err="failed to get container status \"9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35\": rpc error: code = NotFound desc = could not find container \"9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35\": container with ID starting with 9b7a6c940e02f7cc38d01a09292c27b390eb57a181a4d8bfbda50a8d1e342d35 not found: ID does not exist" Oct 30 22:20:16.705995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.705976 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume\") pod \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " Oct 30 22:20:16.706059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.706049 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7\" (UniqueName: \"kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7\") pod \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\" (UID: \"bbc91881-ca67-4d2a-94ed-a67550fe2f5d\") " Oct 30 22:20:16.706293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.706276 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume" pod "bbc91881-ca67-4d2a-94ed-a67550fe2f5d" (UID: "bbc91881-ca67-4d2a-94ed-a67550fe2f5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:16.706470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.706456 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.708144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.708124 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7" pod "bbc91881-ca67-4d2a-94ed-a67550fe2f5d" (UID: "bbc91881-ca67-4d2a-94ed-a67550fe2f5d"). InnerVolumeSpecName "kube-api-access-dx5d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:16.808089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.808059 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5d7\" (UniqueName: \"kubernetes.io/projected/bbc91881-ca67-4d2a-94ed-a67550fe2f5d-kube-api-access-dx5d7\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.923437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:16.923153 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" podStartSLOduration=2.923134091 podStartE2EDuration="2.923134091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:16.921450259 +0000 UTC m=+1996.322006382" watchObservedRunningTime="2026-10-30 22:20:16.923134091 +0000 UTC m=+1996.323690212" Oct 30 22:20:17.112639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.112602 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t"] Oct 30 22:20:17.116467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.116441 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4576d9-5d7b588954-q5f9t"] Oct 30 22:20:17.501459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.501269 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks"] Oct 30 22:20:17.502407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.502380 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bbc91881-ca67-4d2a-94ed-a67550fe2f5d" containerName="payload-processing" Oct 30 22:20:17.502407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.502409 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc91881-ca67-4d2a-94ed-a67550fe2f5d" containerName="payload-processing" Oct 30 22:20:17.502603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.502585 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc91881-ca67-4d2a-94ed-a67550fe2f5d" containerName="payload-processing" Oct 30 22:20:17.507455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.506548 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.561063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.561014 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7350bc-serving-cert\"" Oct 30 22:20:17.564678 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.564638 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks"] Oct 30 22:20:17.631312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.631281 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.631488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.631361 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4p24\" (UniqueName: \"kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.631569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.631504 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.631569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.631520 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.733697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.733659 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4p24\" (UniqueName: \"kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.734032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.733848 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.734032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.733878 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.734032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.734006 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.734362 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:17.734313 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-serving-cert: secret "maas-api-e2e-ipp-b-7350bc-serving-cert" not found Oct 30 22:20:17.734571 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:17.734400 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls podName:7fa24579-ce10-4820-8db8-f11a8f4e7d59 nodeName:}" failed. No retries permitted until 2026-10-30 22:20:18.234378622 +0000 UTC m=+1997.634934730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls") pod "maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" (UID: "7fa24579-ce10-4820-8db8-f11a8f4e7d59") : secret "maas-api-e2e-ipp-b-7350bc-serving-cert" not found Oct 30 22:20:17.735542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.735093 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.754055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.747587 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:17.800271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:17.800230 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4p24\" (UniqueName: \"kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:18.240043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.240004 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:18.242514 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.242489 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:18.426117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.426085 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:18.621136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.621106 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr"] Oct 30 22:20:18.633632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.633613 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.688369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.688326 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a3357-dockercfg-l9cnh\"" Oct 30 22:20:18.689360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.689332 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr"] Oct 30 22:20:18.743976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.740236 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a3357\"" Oct 30 22:20:18.769029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.760747 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.769029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.760874 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjgh\" (UniqueName: \"kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.825889 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:18.825860 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa24579_ce10_4820_8db8_f11a8f4e7d59.slice/crio-e44de4584d431179459a20103f452bd9260bd69b2e9c2c10807a8ca28ed935ea WatchSource:0}: Error finding container e44de4584d431179459a20103f452bd9260bd69b2e9c2c10807a8ca28ed935ea: Status 404 returned error can't find the container with id e44de4584d431179459a20103f452bd9260bd69b2e9c2c10807a8ca28ed935ea Oct 30 22:20:18.833665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.833384 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks"] Oct 30 22:20:18.862813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.862787 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.862919 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.862860 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjgh\" (UniqueName: \"kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.863455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.863436 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.870188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.870166 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc91881-ca67-4d2a-94ed-a67550fe2f5d" path="/var/lib/kubelet/pods/bbc91881-ca67-4d2a-94ed-a67550fe2f5d/volumes" Oct 30 22:20:18.935020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.934986 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjgh\" (UniqueName: \"kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh\") pod \"payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:18.947354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:18.947332 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:19.320215 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:19.320184 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc4c96a_247d_42f3_9b84_59a735032aeb.slice/crio-f3efd5e86b0e875e258ac1123b7e9a6db02691b589d8f63d9d43ad37463761f1 WatchSource:0}: Error finding container f3efd5e86b0e875e258ac1123b7e9a6db02691b589d8f63d9d43ad37463761f1: Status 404 returned error can't find the container with id f3efd5e86b0e875e258ac1123b7e9a6db02691b589d8f63d9d43ad37463761f1 Oct 30 22:20:19.322288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:19.322248 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr"] Oct 30 22:20:19.717189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:19.717152 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" event={"ID":"7fa24579-ce10-4820-8db8-f11a8f4e7d59","Type":"ContainerStarted","Data":"e44de4584d431179459a20103f452bd9260bd69b2e9c2c10807a8ca28ed935ea"} Oct 30 22:20:19.719024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:19.718988 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" event={"ID":"2bc4c96a-247d-42f3-9b84-59a735032aeb","Type":"ContainerStarted","Data":"5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab"} Oct 30 22:20:19.719191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:19.719042 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" event={"ID":"2bc4c96a-247d-42f3-9b84-59a735032aeb","Type":"ContainerStarted","Data":"f3efd5e86b0e875e258ac1123b7e9a6db02691b589d8f63d9d43ad37463761f1"} Oct 30 22:20:19.719191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:19.719085 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:19.841471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:19.841382 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" podStartSLOduration=1.8413659999999998 podStartE2EDuration="1.841366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:19.839223997 +0000 UTC m=+1999.239780117" watchObservedRunningTime="2026-10-30 22:20:19.841366 +0000 UTC m=+1999.241922121" Oct 30 22:20:20.728761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:20.728720 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" event={"ID":"7fa24579-ce10-4820-8db8-f11a8f4e7d59","Type":"ContainerStarted","Data":"6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a"} Oct 30 22:20:20.729297 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:20.728794 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:20.853288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:20.853237 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" podStartSLOduration=2.828201113 podStartE2EDuration="3.853216887s" totalImagesPullingTime="1.025015774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:20.850894702 +0000 UTC m=+2000.251450823" watchObservedRunningTime="2026-10-30 22:20:20.853216887 +0000 UTC m=+2000.253773008" Oct 30 22:20:22.863770 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:22.863737 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:20:26.741455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:26.741428 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:20:27.702665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:27.702637 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:20:30.732412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:30.732381 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:20:31.468934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.468906 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq"] Oct 30 22:20:31.469150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.469113 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" podUID="60619e7b-3510-4efb-b180-49f7804ce2af" containerName="maas-api" containerID="cri-o://1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2" gracePeriod=30 Oct 30 22:20:31.747037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.747009 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:20:31.800669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.800631 5607 generic.go:356] "Generic (PLEG): container finished" podID="60619e7b-3510-4efb-b180-49f7804ce2af" containerID="1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2" exitCode=0 Oct 30 22:20:31.800856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.800722 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" Oct 30 22:20:31.800939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.800871 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" event={"ID":"60619e7b-3510-4efb-b180-49f7804ce2af","Type":"ContainerDied","Data":"1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2"} Oct 30 22:20:31.801042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.800982 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq" event={"ID":"60619e7b-3510-4efb-b180-49f7804ce2af","Type":"ContainerDied","Data":"7b348a4fbcc5760acbb8d47d03ce4c99681d02cb9e8d9e055dc2feee8237cf65"} Oct 30 22:20:31.801042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.801021 5607 scope.go:122] "RemoveContainer" containerID="1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2" Oct 30 22:20:31.812855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.812835 5607 scope.go:122] "RemoveContainer" containerID="1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2" Oct 30 22:20:31.813144 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:31.813122 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2\": container with ID starting with 1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2 not found: ID does not exist" containerID="1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2" Oct 30 22:20:31.813251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.813151 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2"} err="failed to get container status \"1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2\": rpc error: code = NotFound desc = could not find container \"1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2\": container with ID starting with 1d51855c9f778bcfdf2e1aca8bae8ae846172df03e537b8cfdbdf1dd9cc5e2e2 not found: ID does not exist" Oct 30 22:20:31.823783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.823763 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\") pod \"60619e7b-3510-4efb-b180-49f7804ce2af\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " Oct 30 22:20:31.823897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.823879 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca\") pod \"60619e7b-3510-4efb-b180-49f7804ce2af\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " Oct 30 22:20:31.823963 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.823930 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4\" (UniqueName: \"kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4\") pod \"60619e7b-3510-4efb-b180-49f7804ce2af\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " Oct 30 22:20:31.824025 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.824005 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs\") pod \"60619e7b-3510-4efb-b180-49f7804ce2af\" (UID: \"60619e7b-3510-4efb-b180-49f7804ce2af\") " Oct 30 22:20:31.824304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.824273 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca" pod "60619e7b-3510-4efb-b180-49f7804ce2af" (UID: "60619e7b-3510-4efb-b180-49f7804ce2af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:31.824553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.824520 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60619e7b-3510-4efb-b180-49f7804ce2af-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.826168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.826136 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls" pod "60619e7b-3510-4efb-b180-49f7804ce2af" (UID: "60619e7b-3510-4efb-b180-49f7804ce2af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:31.826642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.826618 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs" pod "60619e7b-3510-4efb-b180-49f7804ce2af" (UID: "60619e7b-3510-4efb-b180-49f7804ce2af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:31.826755 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.826690 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4" pod "60619e7b-3510-4efb-b180-49f7804ce2af" (UID: "60619e7b-3510-4efb-b180-49f7804ce2af"). InnerVolumeSpecName "kube-api-access-g5gr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:31.926101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.926075 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.926101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.926100 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gr4\" (UniqueName: \"kubernetes.io/projected/60619e7b-3510-4efb-b180-49f7804ce2af-kube-api-access-g5gr4\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.926317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.926109 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60619e7b-3510-4efb-b180-49f7804ce2af-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:31.996070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.995999 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx"] Oct 30 22:20:31.996247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:31.996224 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" podUID="a48a24c2-390c-468b-8a95-8298bf0d6378" containerName="payload-processing" containerID="cri-o://9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582" gracePeriod=30 Oct 30 22:20:32.047109 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.047083 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv"] Oct 30 22:20:32.240544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.240515 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq"] Oct 30 22:20:32.243747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.243723 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b4b54-6bfc77d96d-fm2nq"] Oct 30 22:20:32.245045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.245028 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:20:32.330487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.330431 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume\") pod \"a48a24c2-390c-468b-8a95-8298bf0d6378\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " Oct 30 22:20:32.330600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.330508 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d\" (UniqueName: \"kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d\") pod \"a48a24c2-390c-468b-8a95-8298bf0d6378\" (UID: \"a48a24c2-390c-468b-8a95-8298bf0d6378\") " Oct 30 22:20:32.330765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.330745 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume" pod "a48a24c2-390c-468b-8a95-8298bf0d6378" (UID: "a48a24c2-390c-468b-8a95-8298bf0d6378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:32.332738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.332718 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d" pod "a48a24c2-390c-468b-8a95-8298bf0d6378" (UID: "a48a24c2-390c-468b-8a95-8298bf0d6378"). InnerVolumeSpecName "kube-api-access-t2t4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:32.431996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.431972 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48a24c2-390c-468b-8a95-8298bf0d6378-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:32.431996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.431996 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2t4d\" (UniqueName: \"kubernetes.io/projected/a48a24c2-390c-468b-8a95-8298bf0d6378-kube-api-access-t2t4d\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:32.808441 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.808405 5607 generic.go:356] "Generic (PLEG): container finished" podID="a48a24c2-390c-468b-8a95-8298bf0d6378" containerID="9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582" exitCode=0 Oct 30 22:20:32.808890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.808447 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" Oct 30 22:20:32.808890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.808489 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" event={"ID":"a48a24c2-390c-468b-8a95-8298bf0d6378","Type":"ContainerDied","Data":"9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582"} Oct 30 22:20:32.808890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.808555 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx" event={"ID":"a48a24c2-390c-468b-8a95-8298bf0d6378","Type":"ContainerDied","Data":"3e11ab78381347c0ce976b4c9e5e9f1f2127a45c1a6def9ac280d9b1865f8547"} Oct 30 22:20:32.808890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.808590 5607 scope.go:122] "RemoveContainer" containerID="9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582" Oct 30 22:20:32.809989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.809791 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" podUID="6c91be69-1499-4b54-ad45-2383cfc4ecba" containerName="payload-pre-processing" containerID="cri-o://87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df" gracePeriod=30 Oct 30 22:20:32.821447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.821428 5607 scope.go:122] "RemoveContainer" containerID="9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582" Oct 30 22:20:32.821773 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:32.821742 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582\": container with ID starting with 9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582 not found: ID does not exist" containerID="9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582" Oct 30 22:20:32.821841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.821774 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582"} err="failed to get container status \"9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582\": rpc error: code = NotFound desc = could not find container \"9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582\": container with ID starting with 9d9bf1f83379dcc064bef84c22b66d64205aed6c0add87b3a4fb681cc486b582 not found: ID does not exist" Oct 30 22:20:32.870872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:32.870848 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60619e7b-3510-4efb-b180-49f7804ce2af" path="/var/lib/kubelet/pods/60619e7b-3510-4efb-b180-49f7804ce2af/volumes" Oct 30 22:20:33.088229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.088206 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:20:33.097126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.097104 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx"] Oct 30 22:20:33.101180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.101141 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b4b54-5b69d6b5fd-r2xjx"] Oct 30 22:20:33.140832 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.140807 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk\" (UniqueName: \"kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk\") pod \"6c91be69-1499-4b54-ad45-2383cfc4ecba\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " Oct 30 22:20:33.141011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.140994 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume\") pod \"6c91be69-1499-4b54-ad45-2383cfc4ecba\" (UID: \"6c91be69-1499-4b54-ad45-2383cfc4ecba\") " Oct 30 22:20:33.141359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.141339 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume" pod "6c91be69-1499-4b54-ad45-2383cfc4ecba" (UID: "6c91be69-1499-4b54-ad45-2383cfc4ecba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:33.141465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.141453 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c91be69-1499-4b54-ad45-2383cfc4ecba-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:33.143036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.143008 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk" pod "6c91be69-1499-4b54-ad45-2383cfc4ecba" (UID: "6c91be69-1499-4b54-ad45-2383cfc4ecba"). InnerVolumeSpecName "kube-api-access-tcxqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:33.242727 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.242694 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcxqk\" (UniqueName: \"kubernetes.io/projected/6c91be69-1499-4b54-ad45-2383cfc4ecba-kube-api-access-tcxqk\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:33.818985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.818920 5607 generic.go:356] "Generic (PLEG): container finished" podID="6c91be69-1499-4b54-ad45-2383cfc4ecba" containerID="87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df" exitCode=0 Oct 30 22:20:33.818985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.818988 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" Oct 30 22:20:33.819481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.819005 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" event={"ID":"6c91be69-1499-4b54-ad45-2383cfc4ecba","Type":"ContainerDied","Data":"87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df"} Oct 30 22:20:33.819481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.819079 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv" event={"ID":"6c91be69-1499-4b54-ad45-2383cfc4ecba","Type":"ContainerDied","Data":"95858a1b288d08beec9f385891287692806712bb3431ee327e914c34ac7ca0bc"} Oct 30 22:20:33.819481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.819108 5607 scope.go:122] "RemoveContainer" containerID="87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df" Oct 30 22:20:33.829825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.829807 5607 scope.go:122] "RemoveContainer" containerID="87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df" Oct 30 22:20:33.830110 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:33.830089 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df\": container with ID starting with 87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df not found: ID does not exist" containerID="87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df" Oct 30 22:20:33.830198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.830119 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df"} err="failed to get container status \"87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df\": rpc error: code = NotFound desc = could not find container \"87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df\": container with ID starting with 87224673b5a0b12b2d7d9579a2205a46e8315c4ed49d3dde2a4fb849f1a7b4df not found: ID does not exist" Oct 30 22:20:33.992228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.992207 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv"] Oct 30 22:20:33.997121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:33.997103 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b4b54-58764b79f5-m56jv"] Oct 30 22:20:34.870315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:34.870285 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c91be69-1499-4b54-ad45-2383cfc4ecba" path="/var/lib/kubelet/pods/6c91be69-1499-4b54-ad45-2383cfc4ecba/volumes" Oct 30 22:20:34.870699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:34.870663 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48a24c2-390c-468b-8a95-8298bf0d6378" path="/var/lib/kubelet/pods/a48a24c2-390c-468b-8a95-8298bf0d6378/volumes" Oct 30 22:20:36.793680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.793646 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk"] Oct 30 22:20:36.794267 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794245 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6c91be69-1499-4b54-ad45-2383cfc4ecba" containerName="payload-pre-processing" Oct 30 22:20:36.794267 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794264 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c91be69-1499-4b54-ad45-2383cfc4ecba" containerName="payload-pre-processing" Oct 30 22:20:36.794370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794275 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a48a24c2-390c-468b-8a95-8298bf0d6378" containerName="payload-processing" Oct 30 22:20:36.794370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794280 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48a24c2-390c-468b-8a95-8298bf0d6378" containerName="payload-processing" Oct 30 22:20:36.794370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794301 5607 cpu_manager.go:414] "containerMap: removing container" podUID="60619e7b-3510-4efb-b180-49f7804ce2af" containerName="maas-api" Oct 30 22:20:36.794370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794306 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60619e7b-3510-4efb-b180-49f7804ce2af" containerName="maas-api" Oct 30 22:20:36.794493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794402 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="60619e7b-3510-4efb-b180-49f7804ce2af" containerName="maas-api" Oct 30 22:20:36.794493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794417 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c91be69-1499-4b54-ad45-2383cfc4ecba" containerName="payload-pre-processing" Oct 30 22:20:36.794493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.794423 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48a24c2-390c-468b-8a95-8298bf0d6378" containerName="payload-processing" Oct 30 22:20:36.797891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.797871 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:36.852164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.852134 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3bd7fe\"" Oct 30 22:20:36.883405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.883377 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:36.883591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.883464 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25tw\" (UniqueName: \"kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw\") pod \"payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:36.950985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.950938 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3bd7fe-dockercfg-sh45x\"" Oct 30 22:20:36.960063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.960022 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk"] Oct 30 22:20:36.977647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.977620 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8"] Oct 30 22:20:36.980850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.980831 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:36.984878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.984854 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25tw\" (UniqueName: \"kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw\") pod \"payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:36.985348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.985191 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:36.986020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:36.985997 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:37.010651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.010622 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8"] Oct 30 22:20:37.054283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.054224 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25tw\" (UniqueName: \"kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw\") pod \"payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:37.086746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.086717 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.086924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.086908 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vgp\" (UniqueName: \"kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp\") pod \"payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.111184 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.111161 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:37.188586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.188544 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.188789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.188766 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vgp\" (UniqueName: \"kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp\") pod \"payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.189430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.189396 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.251476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.251414 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vgp\" (UniqueName: \"kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp\") pod \"payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.305525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.305469 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.486350 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:37.486318 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2a7cc6_5af5_4efc_b529_1cfd5c52da0f.slice/crio-a53fafa7fa7a296b42caba26ec517d17744283b95b8c9b2d4b251dcc814d4afc WatchSource:0}: Error finding container a53fafa7fa7a296b42caba26ec517d17744283b95b8c9b2d4b251dcc814d4afc: Status 404 returned error can't find the container with id a53fafa7fa7a296b42caba26ec517d17744283b95b8c9b2d4b251dcc814d4afc Oct 30 22:20:37.488885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.488863 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk"] Oct 30 22:20:37.682355 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:37.682329 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684d0c5a_a014_4672_95be_b2538a7ccd1d.slice/crio-d1fc3ea90c4e1fd689d1d34716c8c04973eedba69f7f71cb583dec287ace0a06 WatchSource:0}: Error finding container d1fc3ea90c4e1fd689d1d34716c8c04973eedba69f7f71cb583dec287ace0a06: Status 404 returned error can't find the container with id d1fc3ea90c4e1fd689d1d34716c8c04973eedba69f7f71cb583dec287ace0a06 Oct 30 22:20:37.682991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.682945 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8"] Oct 30 22:20:37.853609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.853575 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" event={"ID":"684d0c5a-a014-4672-95be-b2538a7ccd1d","Type":"ContainerStarted","Data":"7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc"} Oct 30 22:20:37.854038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.853622 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" event={"ID":"684d0c5a-a014-4672-95be-b2538a7ccd1d","Type":"ContainerStarted","Data":"d1fc3ea90c4e1fd689d1d34716c8c04973eedba69f7f71cb583dec287ace0a06"} Oct 30 22:20:37.854038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.853649 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:37.855243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.855222 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" event={"ID":"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f","Type":"ContainerStarted","Data":"74a65210761dc8abffd65d814a8f5a8cd301bdb8368ff076562e46ba180c2fd2"} Oct 30 22:20:37.855340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.855260 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" event={"ID":"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f","Type":"ContainerStarted","Data":"a53fafa7fa7a296b42caba26ec517d17744283b95b8c9b2d4b251dcc814d4afc"} Oct 30 22:20:37.855340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.855282 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:37.974593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:37.974555 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" podStartSLOduration=1.974542201 podStartE2EDuration="1.974542201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:37.971697211 +0000 UTC m=+2017.372253333" watchObservedRunningTime="2026-10-30 22:20:37.974542201 +0000 UTC m=+2017.375098321" Oct 30 22:20:38.095975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:38.092806 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" podStartSLOduration=2.092785558 podStartE2EDuration="2.092785558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:38.089629165 +0000 UTC m=+2017.490185286" watchObservedRunningTime="2026-10-30 22:20:38.092785558 +0000 UTC m=+2017.493341696" Oct 30 22:20:43.948719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:43.948686 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8"] Oct 30 22:20:43.951743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:43.951725 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.017152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.017128 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-40fb42-openshift-default-dockercfg-wc8t6\"" Oct 30 22:20:44.059105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.059080 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8"] Oct 30 22:20:44.082556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.082530 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.082697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.082571 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.082697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.082631 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.083013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.082987 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhpg\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.083098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.083045 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.083214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.083178 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.083291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.083230 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.083361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.083344 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.083416 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.083402 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.185610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.185556 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.185802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.185629 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.185802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.185719 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.185802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.185774 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.185860 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.185887 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186008 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186026 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186054 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhpg\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186094 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186496 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186669 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.186792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.186765 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.187273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.187253 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.198213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.198184 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.198801 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.198753 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.249481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.249441 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhpg\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.249619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.249457 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token\") pod \"e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.265837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.265798 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:44.687356 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:44.687322 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144cd0b3_1530_4807_b04f_230c95bbce81.slice/crio-970bb4417c529cb3997bceb954a89ffd9654f0744906db21fe8403aa9a3ae02d WatchSource:0}: Error finding container 970bb4417c529cb3997bceb954a89ffd9654f0744906db21fe8403aa9a3ae02d: Status 404 returned error can't find the container with id 970bb4417c529cb3997bceb954a89ffd9654f0744906db21fe8403aa9a3ae02d Oct 30 22:20:44.690149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.690106 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:44.690268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.690198 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:44.690268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.690243 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:44.691137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.691118 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8"] Oct 30 22:20:44.897437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.897406 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" event={"ID":"144cd0b3-1530-4807-b04f-230c95bbce81","Type":"ContainerStarted","Data":"c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b"} Oct 30 22:20:44.897611 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:44.897463 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" event={"ID":"144cd0b3-1530-4807-b04f-230c95bbce81","Type":"ContainerStarted","Data":"970bb4417c529cb3997bceb954a89ffd9654f0744906db21fe8403aa9a3ae02d"} Oct 30 22:20:45.023465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:45.023416 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podStartSLOduration=2.023402375 podStartE2EDuration="2.023402375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:45.020989943 +0000 UTC m=+2024.421546061" watchObservedRunningTime="2026-10-30 22:20:45.023402375 +0000 UTC m=+2024.423958540" Oct 30 22:20:45.266978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:45.266485 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:45.276650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:45.276570 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:45.903514 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:45.903478 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:45.904399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:45.904372 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:20:48.872075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:48.872047 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:20:48.872656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:48.872105 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:20:51.662746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.662708 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w"] Oct 30 22:20:51.670928 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.670905 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.735977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.731302 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-40fb42\"" Oct 30 22:20:51.751980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.738253 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w"] Oct 30 22:20:51.777756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.777728 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-40fb42-dockercfg-z2p2v\"" Oct 30 22:20:51.779354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.779327 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.779597 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.779508 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvkw\" (UniqueName: \"kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw\") pod \"payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.847127 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.847100 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss"] Oct 30 22:20:51.850189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.850169 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:51.880484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.880461 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.880646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.880551 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvkw\" (UniqueName: \"kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw\") pod \"payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.880646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.880606 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbxb\" (UniqueName: \"kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb\") pod \"payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:51.880781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.880764 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:51.881293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.881273 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.930016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.929887 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss"] Oct 30 22:20:51.949174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.949147 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvkw\" (UniqueName: \"kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw\") pod \"payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:51.982262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.982230 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbxb\" (UniqueName: \"kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb\") pod \"payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:51.982409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.982317 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:51.983237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:51.983213 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:52.006137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.006116 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:52.062275 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.055091 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbxb\" (UniqueName: \"kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb\") pod \"payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:52.163467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.163432 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:52.433860 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:20:52.433824 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e471e0_0951_4192_86d4_dce93dfa8be1.slice/crio-01c55cf4e66cd7fa5867b55b6ec35c304e01703fddcdb4aa4da6a66838e74173 WatchSource:0}: Error finding container 01c55cf4e66cd7fa5867b55b6ec35c304e01703fddcdb4aa4da6a66838e74173: Status 404 returned error can't find the container with id 01c55cf4e66cd7fa5867b55b6ec35c304e01703fddcdb4aa4da6a66838e74173 Oct 30 22:20:52.436599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.436577 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w"] Oct 30 22:20:52.542028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.541997 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss"] Oct 30 22:20:52.863165 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:20:52.863077 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:20:52.961829 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.961785 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" event={"ID":"c0a9fa25-6aaa-4416-a247-e1e92edd51e3","Type":"ContainerStarted","Data":"a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60"} Oct 30 22:20:52.962034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.961841 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" event={"ID":"c0a9fa25-6aaa-4416-a247-e1e92edd51e3","Type":"ContainerStarted","Data":"901af9e3f5aceed3051dd2b8bed1696cba04bda87f96af741ca42ba13cbb888c"} Oct 30 22:20:52.962295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.962274 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:20:52.964085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.964059 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" event={"ID":"c3e471e0-0951-4192-86d4-dce93dfa8be1","Type":"ContainerStarted","Data":"b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58"} Oct 30 22:20:52.964213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.964099 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" event={"ID":"c3e471e0-0951-4192-86d4-dce93dfa8be1","Type":"ContainerStarted","Data":"01c55cf4e66cd7fa5867b55b6ec35c304e01703fddcdb4aa4da6a66838e74173"} Oct 30 22:20:52.964213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:52.964205 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:20:53.081505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:20:53.081446 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" podStartSLOduration=2.081428847 podStartE2EDuration="2.081428847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:53.080459753 +0000 UTC m=+2032.481015875" watchObservedRunningTime="2026-10-30 22:20:53.081428847 +0000 UTC m=+2032.481984969" Oct 30 22:21:03.974312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:03.974278 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:21:03.974783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:03.974357 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:21:04.094724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:04.094674 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" podStartSLOduration=13.094660014 podStartE2EDuration="13.094660014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:53.204355186 +0000 UTC m=+2032.604911334" watchObservedRunningTime="2026-10-30 22:21:04.094660014 +0000 UTC m=+2043.495216135" Oct 30 22:21:21.515575 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.515541 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc"] Oct 30 22:21:21.520311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.520290 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.579992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.579932 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc"] Oct 30 22:21:21.598091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.592134 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dqz\" (UniqueName: \"kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.598091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.592248 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.629280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.629243 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-872f4a\"" Oct 30 22:21:21.629515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.629386 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-872f4a-dockercfg-r6bcr\"" Oct 30 22:21:21.693283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.693253 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dqz\" (UniqueName: \"kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.693480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.693347 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.693480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.693370 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq"] Oct 30 22:21:21.694449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.694418 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.698534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.698506 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:21.726152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.726118 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq"] Oct 30 22:21:21.758031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.757998 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dqz\" (UniqueName: \"kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz\") pod \"payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.794650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.794572 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:21.794813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.794736 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7l7g\" (UniqueName: \"kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g\") pod \"payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:21.840084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.840049 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:21.896661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.896625 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:21.896944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.896741 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7l7g\" (UniqueName: \"kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g\") pod \"payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:21.898359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.898116 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:21.961607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:21.961542 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7l7g\" (UniqueName: \"kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g\") pod \"payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:22.012929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:22.012894 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:22.222908 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:21:22.222878 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ce56c8_4286_46ac_8396_02cca176bed1.slice/crio-e58552bbe3c949f1ebe22f3edc2d1d38959bc4d4073b2ff2bc85b5f26ae7fbb9 WatchSource:0}: Error finding container e58552bbe3c949f1ebe22f3edc2d1d38959bc4d4073b2ff2bc85b5f26ae7fbb9: Status 404 returned error can't find the container with id e58552bbe3c949f1ebe22f3edc2d1d38959bc4d4073b2ff2bc85b5f26ae7fbb9 Oct 30 22:21:22.225625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:22.225601 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc"] Oct 30 22:21:22.361941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:22.361871 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl"] Oct 30 22:21:22.396507 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:21:22.396477 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9334a5f_3280_4adc_abb7_63c75f0af9cb.slice/crio-b11c0733e931a8c8bb66b3060762ccfed32da6d64f70789ba8c237d787e043b4 WatchSource:0}: Error finding container b11c0733e931a8c8bb66b3060762ccfed32da6d64f70789ba8c237d787e043b4: Status 404 returned error can't find the container with id b11c0733e931a8c8bb66b3060762ccfed32da6d64f70789ba8c237d787e043b4 Oct 30 22:21:22.398451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:22.398429 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq"] Oct 30 22:21:22.868848 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:21:22.868819 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:21:23.189405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.189357 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" event={"ID":"f9334a5f-3280-4adc-abb7-63c75f0af9cb","Type":"ContainerStarted","Data":"4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2"} Oct 30 22:21:23.189594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.189425 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" event={"ID":"f9334a5f-3280-4adc-abb7-63c75f0af9cb","Type":"ContainerStarted","Data":"b11c0733e931a8c8bb66b3060762ccfed32da6d64f70789ba8c237d787e043b4"} Oct 30 22:21:23.189594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.189560 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:23.191242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.191217 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" event={"ID":"c8ce56c8-4286-46ac-8396-02cca176bed1","Type":"ContainerStarted","Data":"de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73"} Oct 30 22:21:23.191395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.191262 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" event={"ID":"c8ce56c8-4286-46ac-8396-02cca176bed1","Type":"ContainerStarted","Data":"e58552bbe3c949f1ebe22f3edc2d1d38959bc4d4073b2ff2bc85b5f26ae7fbb9"} Oct 30 22:21:23.191395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.191322 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:23.308927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.308874 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" podStartSLOduration=2.308859399 podStartE2EDuration="2.308859399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.307925855 +0000 UTC m=+2062.708481974" watchObservedRunningTime="2026-10-30 22:21:23.308859399 +0000 UTC m=+2062.709415564" Oct 30 22:21:23.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:23.428782 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" podStartSLOduration=2.428756108 podStartE2EDuration="2.428756108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.425895675 +0000 UTC m=+2062.826451797" watchObservedRunningTime="2026-10-30 22:21:23.428756108 +0000 UTC m=+2062.829312221" Oct 30 22:21:24.199457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:24.199426 5607 generic.go:356] "Generic (PLEG): container finished" podID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerID="de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078" exitCode=0 Oct 30 22:21:24.199930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:24.199500 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" event={"ID":"9f264482-3845-4221-bbf1-95a1b3b386ae","Type":"ContainerDied","Data":"de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078"} Oct 30 22:21:24.200020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:24.199982 5607 scope.go:122] "RemoveContainer" containerID="de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078" Oct 30 22:21:24.201713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:24.201687 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:21:24.201792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:24.201777 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:21:24.201830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:24.201816 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:21:25.168653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:25.168604 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:21:25.225187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:25.225148 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" event={"ID":"9f264482-3845-4221-bbf1-95a1b3b386ae","Type":"ContainerStarted","Data":"30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1"} Oct 30 22:21:25.252505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:25.252471 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:21:25.253800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:25.253770 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:21:25.253966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:25.253842 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:21:26.252853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:26.252818 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:21:26.253328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:26.252866 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:21:27.253040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.253007 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:21:27.253543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.253055 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:21:27.363219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.363160 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" podUID="43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" containerName="payload-pre-processing" containerID="cri-o://5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986" gracePeriod=25 Oct 30 22:21:27.678425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.678400 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:21:27.783196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.783166 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms\" (UniqueName: \"kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms\") pod \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " Oct 30 22:21:27.783397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.783382 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume\") pod \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\" (UID: \"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41\") " Oct 30 22:21:27.783806 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.783784 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume" pod "43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" (UID: "43ce887c-6d7e-4ea3-85b2-9e8c245a1e41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:27.785447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.785425 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms" pod "43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" (UID: "43ce887c-6d7e-4ea3-85b2-9e8c245a1e41"). InnerVolumeSpecName "kube-api-access-mf8ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:27.884804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.884772 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:27.884804 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:27.884800 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf8ms\" (UniqueName: \"kubernetes.io/projected/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41-kube-api-access-mf8ms\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.246571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.246540 5607 generic.go:356] "Generic (PLEG): container finished" podID="43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" containerID="5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986" exitCode=0 Oct 30 22:21:28.246786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.246589 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" Oct 30 22:21:28.246786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.246589 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" event={"ID":"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41","Type":"ContainerDied","Data":"5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986"} Oct 30 22:21:28.246786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.246723 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl" event={"ID":"43ce887c-6d7e-4ea3-85b2-9e8c245a1e41","Type":"ContainerDied","Data":"a41738110f23fc81e8bbbea7826692de1eb050541a6222ed9eda0de3b7cfb974"} Oct 30 22:21:28.246786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.246759 5607 scope.go:122] "RemoveContainer" containerID="5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986" Oct 30 22:21:28.252212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.252186 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:21:28.252352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.252240 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:21:28.257005 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.256986 5607 scope.go:122] "RemoveContainer" containerID="5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986" Oct 30 22:21:28.257321 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:21:28.257258 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986\": container with ID starting with 5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986 not found: ID does not exist" containerID="5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986" Oct 30 22:21:28.257321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.257280 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986"} err="failed to get container status \"5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986\": rpc error: code = NotFound desc = could not find container \"5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986\": container with ID starting with 5e85f8c79ea4ff78c31cc73818286246b334cdf66062033ede72af982cc82986 not found: ID does not exist" Oct 30 22:21:28.424010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.423971 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl"] Oct 30 22:21:28.426477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.426457 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7350bc-7467486df5-6nvkl"] Oct 30 22:21:28.873573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:28.873531 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" path="/var/lib/kubelet/pods/43ce887c-6d7e-4ea3-85b2-9e8c245a1e41/volumes" Oct 30 22:21:29.253186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:29.253149 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:21:29.253383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:29.253220 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:21:30.252439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:30.252401 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 22:21:30.252941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:30.252456 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 22:21:31.256494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:31.256461 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:21:31.256937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:31.256911 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:21:31.257620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:31.257603 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:21:34.202877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:34.202843 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:21:34.203357 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:34.202913 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:21:43.483662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.483576 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg"] Oct 30 22:21:43.484249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.484234 5607 cpu_manager.go:414] "containerMap: removing container" podUID="43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" containerName="payload-pre-processing" Oct 30 22:21:43.484296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.484250 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" containerName="payload-pre-processing" Oct 30 22:21:43.484395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.484385 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ce887c-6d7e-4ea3-85b2-9e8c245a1e41" containerName="payload-pre-processing" Oct 30 22:21:43.487501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.487480 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.541085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.541059 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-40fb42-serving-cert\"" Oct 30 22:21:43.588284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.588254 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.588448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.588291 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.588505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.588481 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.588601 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.588586 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88wq\" (UniqueName: \"kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.595262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.595239 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg"] Oct 30 22:21:43.689360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.689325 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.689570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.689373 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88wq\" (UniqueName: \"kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.689570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.689476 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.689570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.689497 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.689754 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:21:43.689609 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-serving-cert: secret "maas-api-e2e-praxis-a-40fb42-serving-cert" not found Oct 30 22:21:43.689754 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:21:43.689682 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls podName:2740c3d2-b33f-4e20-9deb-6538a4e9413f nodeName:}" failed. No retries permitted until 2026-10-30 22:21:44.189666248 +0000 UTC m=+2083.590222347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls") pod "maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" (UID: "2740c3d2-b33f-4e20-9deb-6538a4e9413f") : secret "maas-api-e2e-praxis-a-40fb42-serving-cert" not found Oct 30 22:21:43.690134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.690116 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.692157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.692135 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:43.753886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:43.753807 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88wq\" (UniqueName: \"kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:44.195194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:44.195166 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:44.197742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:44.197719 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:44.401462 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:44.401431 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:44.775229 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:21:44.775181 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2740c3d2_b33f_4e20_9deb_6538a4e9413f.slice/crio-3294ba272fdfffcd4117bcdbf15997cc3d59a2aea70a2e5453d2162cf1598d78 WatchSource:0}: Error finding container 3294ba272fdfffcd4117bcdbf15997cc3d59a2aea70a2e5453d2162cf1598d78: Status 404 returned error can't find the container with id 3294ba272fdfffcd4117bcdbf15997cc3d59a2aea70a2e5453d2162cf1598d78 Oct 30 22:21:44.776863 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:44.776840 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:21:44.778072 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:44.778049 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg"] Oct 30 22:21:45.359557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:45.359518 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" event={"ID":"2740c3d2-b33f-4e20-9deb-6538a4e9413f","Type":"ContainerStarted","Data":"3294ba272fdfffcd4117bcdbf15997cc3d59a2aea70a2e5453d2162cf1598d78"} Oct 30 22:21:46.372670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:46.372605 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" event={"ID":"2740c3d2-b33f-4e20-9deb-6538a4e9413f","Type":"ContainerStarted","Data":"9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a"} Oct 30 22:21:46.375120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:46.375095 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:46.496186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:46.496134 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" podStartSLOduration=2.623902623 podStartE2EDuration="3.496119209s" totalImagesPullingTime="872.216586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:46.494697429 +0000 UTC m=+2085.895253551" watchObservedRunningTime="2026-10-30 22:21:46.496119209 +0000 UTC m=+2085.896675331" Oct 30 22:21:52.863326 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:21:52.863283 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:21:53.393604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:53.393575 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:21:59.120079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.119908 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l"] Oct 30 22:21:59.120620 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.120195 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" podUID="ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" containerName="payload-pre-processing" containerID="cri-o://3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29" gracePeriod=30 Oct 30 22:21:59.402230 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.402202 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:21:59.457813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.457785 5607 generic.go:356] "Generic (PLEG): container finished" podID="ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" containerID="3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29" exitCode=0 Oct 30 22:21:59.457968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.457832 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" Oct 30 22:21:59.457968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.457875 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" event={"ID":"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa","Type":"ContainerDied","Data":"3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29"} Oct 30 22:21:59.457968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.457923 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l" event={"ID":"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa","Type":"ContainerDied","Data":"1d407815dd050e9ac710541811e7fd0c5a0824918eef39cd7ee327ef7d2ef3fd"} Oct 30 22:21:59.457968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.457946 5607 scope.go:122] "RemoveContainer" containerID="3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29" Oct 30 22:21:59.467362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.467343 5607 scope.go:122] "RemoveContainer" containerID="3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29" Oct 30 22:21:59.467622 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:21:59.467603 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29\": container with ID starting with 3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29 not found: ID does not exist" containerID="3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29" Oct 30 22:21:59.467677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.467630 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29"} err="failed to get container status \"3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29\": rpc error: code = NotFound desc = could not find container \"3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29\": container with ID starting with 3e514ec9956f8a759d35874ba0ff6900b8f4cc3ccd8d9735bcca681c22973a29 not found: ID does not exist" Oct 30 22:21:59.482180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.482161 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2\" (UniqueName: \"kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2\") pod \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " Oct 30 22:21:59.482272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.482230 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume\") pod \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\" (UID: \"ecfdb567-f0f2-4e68-985a-fc5e9603bbfa\") " Oct 30 22:21:59.482562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.482542 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume" pod "ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" (UID: "ecfdb567-f0f2-4e68-985a-fc5e9603bbfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:59.484398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.484378 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2" pod "ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" (UID: "ecfdb567-f0f2-4e68-985a-fc5e9603bbfa"). InnerVolumeSpecName "kube-api-access-7jzm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:59.635029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.630883 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzm2\" (UniqueName: \"kubernetes.io/projected/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-kube-api-access-7jzm2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.635029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.630918 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:59.881522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.881492 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l"] Oct 30 22:21:59.886939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:21:59.886871 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8daffd-7b886cf7d7-tmr2l"] Oct 30 22:22:00.870165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:00.870134 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" path="/var/lib/kubelet/pods/ecfdb567-f0f2-4e68-985a-fc5e9603bbfa/volumes" Oct 30 22:22:01.169783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:01.169702 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:22:01.170412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:01.170392 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:22:07.588151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:07.584645 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn"] Oct 30 22:22:07.588730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:07.588551 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" containerID="cri-o://69455949d8d8a29c3d573a950d9732740bc18b71f9f5d0979ea919e26e917d75" gracePeriod=30 Oct 30 22:22:10.819004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:10.818838 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr"] Oct 30 22:22:12.566438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.566389 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:22:12.566924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.566461 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:22:12.633506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.633473 5607 generic.go:356] "Generic (PLEG): container finished" podID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerID="69455949d8d8a29c3d573a950d9732740bc18b71f9f5d0979ea919e26e917d75" exitCode=0 Oct 30 22:22:12.633630 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.633547 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" event={"ID":"d42d7934-e97a-45fd-a4b1-58420951e26e","Type":"ContainerDied","Data":"69455949d8d8a29c3d573a950d9732740bc18b71f9f5d0979ea919e26e917d75"} Oct 30 22:22:12.633686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.633671 5607 scope.go:122] "RemoveContainer" containerID="ea4c355f3383c81122f9d1f51d6fd01ef7b690de715fd8a10861f9471dbf829a" Oct 30 22:22:12.821922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.821858 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:22:12.891280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891254 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891298 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891328 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891348 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891403 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891435 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891525 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891540 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:12.891654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891610 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891633 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:12.891925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891687 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy\") pod \"d42d7934-e97a-45fd-a4b1-58420951e26e\" (UID: \"d42d7934-e97a-45fd-a4b1-58420951e26e\") " Oct 30 22:22:12.891925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.891801 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:12.892543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.892113 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:12.892543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.892141 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.892543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.892162 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.892543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.892177 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.892543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.892310 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:12.894586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.894565 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:12.894726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.894700 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:12.894922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.894808 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:12.895090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.895064 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf" pod "d42d7934-e97a-45fd-a4b1-58420951e26e" (UID: "d42d7934-e97a-45fd-a4b1-58420951e26e"). InnerVolumeSpecName "kube-api-access-tx6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:12.993472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.993423 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.993472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.993452 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.993472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.993467 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.993472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.993479 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42d7934-e97a-45fd-a4b1-58420951e26e-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.993835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.993489 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx6cf\" (UniqueName: \"kubernetes.io/projected/d42d7934-e97a-45fd-a4b1-58420951e26e-kube-api-access-tx6cf\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:12.993835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:12.993500 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42d7934-e97a-45fd-a4b1-58420951e26e-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:13.641564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:13.641526 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" event={"ID":"d42d7934-e97a-45fd-a4b1-58420951e26e","Type":"ContainerDied","Data":"5a538cafa29adf27f41a2d44854529cca63dbdbaf12b334fb94abdb19644b736"} Oct 30 22:22:13.642097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:13.641576 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn" Oct 30 22:22:13.642097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:13.641599 5607 scope.go:122] "RemoveContainer" containerID="69455949d8d8a29c3d573a950d9732740bc18b71f9f5d0979ea919e26e917d75" Oct 30 22:22:13.862786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:13.862750 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn"] Oct 30 22:22:13.865372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:13.865351 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8daffd-openshift-default-68ffdbcf7c-88lkn"] Oct 30 22:22:14.870805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:14.870767 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" path="/var/lib/kubelet/pods/d42d7934-e97a-45fd-a4b1-58420951e26e/volumes" Oct 30 22:22:15.656368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.656341 5607 generic.go:356] "Generic (PLEG): container finished" podID="144cd0b3-1530-4807-b04f-230c95bbce81" containerID="c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b" exitCode=0 Oct 30 22:22:15.656507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.656402 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" event={"ID":"144cd0b3-1530-4807-b04f-230c95bbce81","Type":"ContainerDied","Data":"c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b"} Oct 30 22:22:15.656773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.656759 5607 scope.go:122] "RemoveContainer" containerID="c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b" Oct 30 22:22:15.658390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.658358 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:15.658502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.658444 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:15.658502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.658494 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:15.820473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.820395 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" podUID="2bc4c96a-247d-42f3-9b84-59a735032aeb" containerName="payload-pre-processing" containerID="cri-o://5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab" gracePeriod=25 Oct 30 22:22:15.904450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:15.904360 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:22:16.124451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.124416 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:22:16.236905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.236875 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh\" (UniqueName: \"kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh\") pod \"2bc4c96a-247d-42f3-9b84-59a735032aeb\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " Oct 30 22:22:16.237125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.237005 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume\") pod \"2bc4c96a-247d-42f3-9b84-59a735032aeb\" (UID: \"2bc4c96a-247d-42f3-9b84-59a735032aeb\") " Oct 30 22:22:16.237360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.237337 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume" pod "2bc4c96a-247d-42f3-9b84-59a735032aeb" (UID: "2bc4c96a-247d-42f3-9b84-59a735032aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:16.239476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.239452 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh" pod "2bc4c96a-247d-42f3-9b84-59a735032aeb" (UID: "2bc4c96a-247d-42f3-9b84-59a735032aeb"). InnerVolumeSpecName "kube-api-access-xgjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:16.338615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.338575 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjgh\" (UniqueName: \"kubernetes.io/projected/2bc4c96a-247d-42f3-9b84-59a735032aeb-kube-api-access-xgjgh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:16.338615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.338611 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc4c96a-247d-42f3-9b84-59a735032aeb-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:16.665536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.665499 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" event={"ID":"144cd0b3-1530-4807-b04f-230c95bbce81","Type":"ContainerStarted","Data":"42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8"} Oct 30 22:22:16.667006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.666981 5607 generic.go:356] "Generic (PLEG): container finished" podID="2bc4c96a-247d-42f3-9b84-59a735032aeb" containerID="5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab" exitCode=0 Oct 30 22:22:16.667136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.667013 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" Oct 30 22:22:16.667136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.667069 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" event={"ID":"2bc4c96a-247d-42f3-9b84-59a735032aeb","Type":"ContainerDied","Data":"5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab"} Oct 30 22:22:16.667136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.667116 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr" event={"ID":"2bc4c96a-247d-42f3-9b84-59a735032aeb","Type":"ContainerDied","Data":"f3efd5e86b0e875e258ac1123b7e9a6db02691b589d8f63d9d43ad37463761f1"} Oct 30 22:22:16.667274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.667139 5607 scope.go:122] "RemoveContainer" containerID="5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab" Oct 30 22:22:16.677490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.677472 5607 scope.go:122] "RemoveContainer" containerID="5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab" Oct 30 22:22:16.677751 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:22:16.677730 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab\": container with ID starting with 5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab not found: ID does not exist" containerID="5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab" Oct 30 22:22:16.677833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.677761 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab"} err="failed to get container status \"5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab\": rpc error: code = NotFound desc = could not find container \"5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab\": container with ID starting with 5019c999c4609d651676e7b514180d2ef76aec507dc4661c925108ca2c2aaeab not found: ID does not exist" Oct 30 22:22:16.904190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.904117 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr"] Oct 30 22:22:16.908112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:16.908086 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a3357-668f4f8d7-vwqtr"] Oct 30 22:22:17.179172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179106 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg"] Oct 30 22:22:17.179699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179686 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" Oct 30 22:22:17.179744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179700 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" Oct 30 22:22:17.179744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179723 5607 cpu_manager.go:414] "containerMap: removing container" podUID="ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" containerName="payload-pre-processing" Oct 30 22:22:17.179744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179728 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" containerName="payload-pre-processing" Oct 30 22:22:17.179837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179749 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2bc4c96a-247d-42f3-9b84-59a735032aeb" containerName="payload-pre-processing" Oct 30 22:22:17.179837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179754 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc4c96a-247d-42f3-9b84-59a735032aeb" containerName="payload-pre-processing" Oct 30 22:22:17.179837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179833 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" Oct 30 22:22:17.179926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179843 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfdb567-f0f2-4e68-985a-fc5e9603bbfa" containerName="payload-pre-processing" Oct 30 22:22:17.179926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179851 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" Oct 30 22:22:17.179926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.179860 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc4c96a-247d-42f3-9b84-59a735032aeb" containerName="payload-pre-processing" Oct 30 22:22:17.191562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.191543 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.245693 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.245660 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-872f4a-serving-cert\"" Oct 30 22:22:17.246401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.246379 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg"] Oct 30 22:22:17.249243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.249205 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.249354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.249247 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.249517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.249489 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.249599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.249541 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8q9\" (UniqueName: \"kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.266372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.266341 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:22:17.267370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.267341 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:22:17.267495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.267406 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:22:17.350832 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.350800 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.350832 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.350835 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.351133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.350999 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.351133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.351025 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8q9\" (UniqueName: \"kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.351255 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:22:17.351136 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-serving-cert: secret "maas-api-e2e-ipp-a-872f4a-serving-cert" not found Oct 30 22:22:17.351255 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:22:17.351215 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls podName:eddab44b-4e7e-4c7c-ae75-fddff06f2fd5 nodeName:}" failed. No retries permitted until 2026-10-30 22:22:17.851193179 +0000 UTC m=+2117.251749278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls") pod "maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" (UID: "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5") : secret "maas-api-e2e-ipp-a-872f4a-serving-cert" not found Oct 30 22:22:17.351591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.351566 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.353480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.353456 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.442974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.442748 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8q9\" (UniqueName: \"kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.859679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.859601 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:17.862229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:17.862209 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:18.113028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:18.112194 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:18.266971 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:18.266925 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:22:18.267145 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:18.266995 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:22:18.490287 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:22:18.490253 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddab44b_4e7e_4c7c_ae75_fddff06f2fd5.slice/crio-32c8ec5fabd6174c3e69575800abddc767356ea89530991cd34213c7c20fac3b WatchSource:0}: Error finding container 32c8ec5fabd6174c3e69575800abddc767356ea89530991cd34213c7c20fac3b: Status 404 returned error can't find the container with id 32c8ec5fabd6174c3e69575800abddc767356ea89530991cd34213c7c20fac3b Oct 30 22:22:18.493691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:18.493587 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg"] Oct 30 22:22:18.681680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:18.681641 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" event={"ID":"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5","Type":"ContainerStarted","Data":"32c8ec5fabd6174c3e69575800abddc767356ea89530991cd34213c7c20fac3b"} Oct 30 22:22:18.869967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:18.869882 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc4c96a-247d-42f3-9b84-59a735032aeb" path="/var/lib/kubelet/pods/2bc4c96a-247d-42f3-9b84-59a735032aeb/volumes" Oct 30 22:22:19.266764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:19.266729 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:22:19.267228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:19.266794 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:22:19.692010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:19.691971 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" event={"ID":"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5","Type":"ContainerStarted","Data":"9fb69e1ce9a7f6323c4e61b2d81d8c4b92c0d7afaf95681ee983815e64089305"} Oct 30 22:22:19.692155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:19.692081 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:19.846774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:19.846664 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" podStartSLOduration=1.790482567 podStartE2EDuration="2.846649265s" totalImagesPullingTime="1.056166698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:19.844827748 +0000 UTC m=+2119.245383869" watchObservedRunningTime="2026-10-30 22:22:19.846649265 +0000 UTC m=+2119.247205386" Oct 30 22:22:20.266218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:20.266172 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:22:20.266377 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:20.266238 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:22:21.266783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:21.266750 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:22:21.267244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:21.266800 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:22:22.266441 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:22.266404 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 22:22:22.266617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:22.266460 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 22:22:22.866214 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:22:22.866170 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:22:23.271082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:23.271052 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:22:23.271378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:23.271350 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:22:23.272150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:23.272129 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:22:25.720876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:25.720846 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:22:44.337015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:44.336972 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58"] Oct 30 22:22:44.337663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:44.337200 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" containerID="cri-o://d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87" gracePeriod=30 Oct 30 22:22:45.667574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.667541 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv"] Oct 30 22:22:45.668255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.668238 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" Oct 30 22:22:45.668305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.668256 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42d7934-e97a-45fd-a4b1-58420951e26e" containerName="istio-proxy" Oct 30 22:22:45.671985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.671936 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.712930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.712905 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.713219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.713191 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5lf\" (UniqueName: \"kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf\") pod \"payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.778731 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.778417 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ebcc6d\"" Oct 30 22:22:45.778731 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.778682 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ebcc6d-dockercfg-sdrcm\"" Oct 30 22:22:45.781859 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.781833 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv"] Oct 30 22:22:45.834976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.834818 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5lf\" (UniqueName: \"kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf\") pod \"payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.834976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.834930 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.841236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.835826 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.841236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.836487 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4"] Oct 30 22:22:45.850017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.843484 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:45.879846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.879805 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4"] Oct 30 22:22:45.907469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.907442 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5lf\" (UniqueName: \"kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf\") pod \"payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:45.941920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.941886 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:45.942090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.942034 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcp8\" (UniqueName: \"kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8\") pod \"payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:45.985777 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:45.985744 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:46.043558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.043520 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:46.043739 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.043593 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcp8\" (UniqueName: \"kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8\") pod \"payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:46.044483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.044281 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:46.106490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.106414 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcp8\" (UniqueName: \"kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8\") pod \"payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:46.172438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.172406 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:46.369861 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:22:46.369812 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1b72d4_0add_4ea4_acae_6a90fbbafce5.slice/crio-19323e829062bf3e7a0809d96f5349f5f5baf7b238af8d3b79b48a9d7fb5b01a WatchSource:0}: Error finding container 19323e829062bf3e7a0809d96f5349f5f5baf7b238af8d3b79b48a9d7fb5b01a: Status 404 returned error can't find the container with id 19323e829062bf3e7a0809d96f5349f5f5baf7b238af8d3b79b48a9d7fb5b01a Oct 30 22:22:46.372791 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.372771 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv"] Oct 30 22:22:46.551230 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:22:46.551201 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e74e7de_63e0_469f_bf3d_0978f98e3182.slice/crio-1f7c72b258b7bb8f08677a2f1e01e79244124b75cf7a9173d891896f603a9b80 WatchSource:0}: Error finding container 1f7c72b258b7bb8f08677a2f1e01e79244124b75cf7a9173d891896f603a9b80: Status 404 returned error can't find the container with id 1f7c72b258b7bb8f08677a2f1e01e79244124b75cf7a9173d891896f603a9b80 Oct 30 22:22:46.554196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.554174 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4"] Oct 30 22:22:46.889751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.889669 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" event={"ID":"0f1b72d4-0add-4ea4-acae-6a90fbbafce5","Type":"ContainerStarted","Data":"cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1"} Oct 30 22:22:46.889751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.889726 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:46.889751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.889745 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" event={"ID":"0f1b72d4-0add-4ea4-acae-6a90fbbafce5","Type":"ContainerStarted","Data":"19323e829062bf3e7a0809d96f5349f5f5baf7b238af8d3b79b48a9d7fb5b01a"} Oct 30 22:22:46.891275 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.891254 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" event={"ID":"7e74e7de-63e0-469f-bf3d-0978f98e3182","Type":"ContainerStarted","Data":"3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c"} Oct 30 22:22:46.891356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.891295 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" event={"ID":"7e74e7de-63e0-469f-bf3d-0978f98e3182","Type":"ContainerStarted","Data":"1f7c72b258b7bb8f08677a2f1e01e79244124b75cf7a9173d891896f603a9b80"} Oct 30 22:22:46.891356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:46.891342 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:47.017140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:47.017092 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" podStartSLOduration=2.017077839 podStartE2EDuration="2.017077839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:47.015752017 +0000 UTC m=+2146.416308224" watchObservedRunningTime="2026-10-30 22:22:47.017077839 +0000 UTC m=+2146.417633959" Oct 30 22:22:47.135073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:47.135013 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" podStartSLOduration=2.134995692 podStartE2EDuration="2.134995692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:47.133382041 +0000 UTC m=+2146.533938165" watchObservedRunningTime="2026-10-30 22:22:47.134995692 +0000 UTC m=+2146.535551815" Oct 30 22:22:49.620123 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.620060 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:22:49.789288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.789258 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.789568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.789536 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.789736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.789709 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.789994 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.789939 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.790164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790031 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:49.790294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790107 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:49.790294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790147 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.790294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790197 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:49.790294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790214 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.790294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790256 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.790562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790323 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.790562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790378 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy\") pod \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\" (UID: \"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a\") " Oct 30 22:22:49.790666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790607 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:49.790718 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.790695 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:49.791194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.791166 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.791302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.791199 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.791302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.791217 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.791302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.791231 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.791302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.791246 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.792058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.792026 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:49.792480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.792456 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:49.792780 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.792760 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "kube-api-access-kjpw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:49.793058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.793039 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy" pod "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" (UID: "421866cc-91cc-411e-a1bc-d6ea9f7e1f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:49.892839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.892770 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.892839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.892796 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.892839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.892811 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjpw4\" (UniqueName: \"kubernetes.io/projected/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-kube-api-access-kjpw4\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.892839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.892824 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:49.912177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.912146 5607 generic.go:356] "Generic (PLEG): container finished" podID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerID="d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87" exitCode=0 Oct 30 22:22:49.912287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.912193 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" Oct 30 22:22:49.912287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.912238 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" event={"ID":"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a","Type":"ContainerDied","Data":"d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87"} Oct 30 22:22:49.912362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.912293 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58" event={"ID":"421866cc-91cc-411e-a1bc-d6ea9f7e1f2a","Type":"ContainerDied","Data":"5949323faec14a99ae45500787babd7988d897b1e200464459c9acaaea7b7a4a"} Oct 30 22:22:49.912362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.912318 5607 scope.go:122] "RemoveContainer" containerID="d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87" Oct 30 22:22:49.926165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.926143 5607 scope.go:122] "RemoveContainer" containerID="d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87" Oct 30 22:22:49.926478 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:22:49.926450 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87\": container with ID starting with d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87 not found: ID does not exist" containerID="d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87" Oct 30 22:22:49.926547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:49.926479 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87"} err="failed to get container status \"d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87\": rpc error: code = NotFound desc = could not find container \"d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87\": container with ID starting with d6e0aaf095cb1418eeaf08d25267ad0079fc940ff6ace2dc07d1e32ffbcbfe87 not found: ID does not exist" Oct 30 22:22:50.089122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:50.089085 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58"] Oct 30 22:22:50.095283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:50.095259 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-v6c58"] Oct 30 22:22:50.871283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:50.871255 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" path="/var/lib/kubelet/pods/421866cc-91cc-411e-a1bc-d6ea9f7e1f2a/volumes" Oct 30 22:22:52.864383 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:22:52.864346 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:22:57.254134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.254099 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6"] Oct 30 22:22:57.254720 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.254707 5607 cpu_manager.go:414] "containerMap: removing container" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" Oct 30 22:22:57.254781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.254721 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" Oct 30 22:22:57.254921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.254876 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="421866cc-91cc-411e-a1bc-d6ea9f7e1f2a" containerName="istio-proxy" Oct 30 22:22:57.258369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.258354 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274519 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmg9\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274566 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274642 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274692 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274742 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274763 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274789 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274805 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.274926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.274914 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.313202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.313174 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-923d78-openshift-default-dockercfg-blpfl\"" Oct 30 22:22:57.359891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.359860 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6"] Oct 30 22:22:57.375876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.375847 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.375993 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmg9\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376058 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376106 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376132 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376187 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376221 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376264 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376288 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376303 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376477 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376674 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.376876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.376750 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.377134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.377106 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.378901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.378880 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.379180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.379160 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.439135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.439109 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.440601 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.440578 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmg9\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9\") pod \"e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.572689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.572598 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:57.901550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.901481 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:22:57.901550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:57.901538 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:22:57.999619 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:22:57.999590 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a07bdb7_9b74_494d_855b_1a3f7d4b9d0d.slice/crio-2f95b206fdea2d4f408c7393d8eb4e58676059ab9f6e9e7b873c3b2e3b4ecf78 WatchSource:0}: Error finding container 2f95b206fdea2d4f408c7393d8eb4e58676059ab9f6e9e7b873c3b2e3b4ecf78: Status 404 returned error can't find the container with id 2f95b206fdea2d4f408c7393d8eb4e58676059ab9f6e9e7b873c3b2e3b4ecf78 Oct 30 22:22:58.002091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:58.002058 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:58.002168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:58.002126 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:58.002168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:58.002156 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:58.002826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:58.002801 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6"] Oct 30 22:22:58.973858 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:58.973822 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" event={"ID":"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d","Type":"ContainerStarted","Data":"10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21"} Oct 30 22:22:58.974325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:58.973883 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" event={"ID":"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d","Type":"ContainerStarted","Data":"2f95b206fdea2d4f408c7393d8eb4e58676059ab9f6e9e7b873c3b2e3b4ecf78"} Oct 30 22:22:59.142718 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:59.142665 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podStartSLOduration=2.142652786 podStartE2EDuration="2.142652786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:59.11505684 +0000 UTC m=+2158.515612961" watchObservedRunningTime="2026-10-30 22:22:59.142652786 +0000 UTC m=+2158.543208906" Oct 30 22:22:59.572867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:59.572836 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:59.578870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:59.578846 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:59.980085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:59.980055 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:22:59.981063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:22:59.981043 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:23:05.740509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.740460 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk"] Oct 30 22:23:05.752429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.752399 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:05.810771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.810729 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk"] Oct 30 22:23:05.855398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.855366 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-923d78\"" Oct 30 22:23:05.855559 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.855418 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-923d78-dockercfg-7rr66\"" Oct 30 22:23:05.885532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.885493 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:05.885713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.885691 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtzr\" (UniqueName: \"kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr\") pod \"payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:05.987031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.987001 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:05.987193 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.987126 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtzr\" (UniqueName: \"kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr\") pod \"payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:05.987974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:05.987932 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:06.049695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:06.049623 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtzr\" (UniqueName: \"kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr\") pod \"payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:06.086246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:06.086224 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:06.480561 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:23:06.480307 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1d28d0_d062_4382_87ea_cdf2059251d0.slice/crio-ca13a70d98ff5fb8f86df6adf3ff8308b8dba2f6eab451de98320f3b80d50153 WatchSource:0}: Error finding container ca13a70d98ff5fb8f86df6adf3ff8308b8dba2f6eab451de98320f3b80d50153: Status 404 returned error can't find the container with id ca13a70d98ff5fb8f86df6adf3ff8308b8dba2f6eab451de98320f3b80d50153 Oct 30 22:23:06.483682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:06.483661 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk"] Oct 30 22:23:07.034885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:07.034845 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" event={"ID":"da1d28d0-d062-4382-87ea-cdf2059251d0","Type":"ContainerStarted","Data":"59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d"} Oct 30 22:23:07.035379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:07.034905 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" event={"ID":"da1d28d0-d062-4382-87ea-cdf2059251d0","Type":"ContainerStarted","Data":"ca13a70d98ff5fb8f86df6adf3ff8308b8dba2f6eab451de98320f3b80d50153"} Oct 30 22:23:07.035379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:07.035061 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:07.157532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:07.157481 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" podStartSLOduration=2.157465565 podStartE2EDuration="2.157465565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.156308944 +0000 UTC m=+2166.556865064" watchObservedRunningTime="2026-10-30 22:23:07.157465565 +0000 UTC m=+2166.558021705" Oct 30 22:23:08.574034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.574002 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks"] Oct 30 22:23:08.574520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.574223 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" podUID="7fa24579-ce10-4820-8db8-f11a8f4e7d59" containerName="maas-api" containerID="cri-o://6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a" gracePeriod=30 Oct 30 22:23:08.852968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.852932 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:23:08.943163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.943137 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs\") pod \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " Oct 30 22:23:08.943296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.943181 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca\") pod \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " Oct 30 22:23:08.943409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.943395 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\") pod \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " Oct 30 22:23:08.943500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.943487 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24\" (UniqueName: \"kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24\") pod \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\" (UID: \"7fa24579-ce10-4820-8db8-f11a8f4e7d59\") " Oct 30 22:23:08.943565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.943506 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca" pod "7fa24579-ce10-4820-8db8-f11a8f4e7d59" (UID: "7fa24579-ce10-4820-8db8-f11a8f4e7d59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:08.943879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.943862 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa24579-ce10-4820-8db8-f11a8f4e7d59-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:08.945437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.945417 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs" pod "7fa24579-ce10-4820-8db8-f11a8f4e7d59" (UID: "7fa24579-ce10-4820-8db8-f11a8f4e7d59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:08.945549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.945471 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24" pod "7fa24579-ce10-4820-8db8-f11a8f4e7d59" (UID: "7fa24579-ce10-4820-8db8-f11a8f4e7d59"). InnerVolumeSpecName "kube-api-access-d4p24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:08.945549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:08.945522 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls" pod "7fa24579-ce10-4820-8db8-f11a8f4e7d59" (UID: "7fa24579-ce10-4820-8db8-f11a8f4e7d59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:09.044673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.044640 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:09.044673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.044673 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4p24\" (UniqueName: \"kubernetes.io/projected/7fa24579-ce10-4820-8db8-f11a8f4e7d59-kube-api-access-d4p24\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:09.044855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.044684 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa24579-ce10-4820-8db8-f11a8f4e7d59-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:09.050914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.050887 5607 generic.go:356] "Generic (PLEG): container finished" podID="7fa24579-ce10-4820-8db8-f11a8f4e7d59" containerID="6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a" exitCode=0 Oct 30 22:23:09.051083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.050940 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" Oct 30 22:23:09.051083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.050994 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" event={"ID":"7fa24579-ce10-4820-8db8-f11a8f4e7d59","Type":"ContainerDied","Data":"6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a"} Oct 30 22:23:09.051083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.051052 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks" event={"ID":"7fa24579-ce10-4820-8db8-f11a8f4e7d59","Type":"ContainerDied","Data":"e44de4584d431179459a20103f452bd9260bd69b2e9c2c10807a8ca28ed935ea"} Oct 30 22:23:09.051083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.051076 5607 scope.go:122] "RemoveContainer" containerID="6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a" Oct 30 22:23:09.062801 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.062784 5607 scope.go:122] "RemoveContainer" containerID="6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a" Oct 30 22:23:09.063103 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:09.063078 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a\": container with ID starting with 6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a not found: ID does not exist" containerID="6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a" Oct 30 22:23:09.063179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.063109 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a"} err="failed to get container status \"6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a\": rpc error: code = NotFound desc = could not find container \"6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a\": container with ID starting with 6d63782b1b7e8b000676affc43b777f3b8d89759eecce1e2379895af2b81596a not found: ID does not exist" Oct 30 22:23:09.223378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.223346 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks"] Oct 30 22:23:09.227860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:09.227836 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7350bc-7db675864f-cdkks"] Oct 30 22:23:10.871323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:10.871248 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa24579-ce10-4820-8db8-f11a8f4e7d59" path="/var/lib/kubelet/pods/7fa24579-ce10-4820-8db8-f11a8f4e7d59/volumes" Oct 30 22:23:18.044637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:18.044607 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:23:18.223389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:18.223360 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw"] Oct 30 22:23:18.223635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:18.223582 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" containerID="cri-o://30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1" gracePeriod=30 Oct 30 22:23:20.461207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.461174 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm"] Oct 30 22:23:20.461891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.461876 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7fa24579-ce10-4820-8db8-f11a8f4e7d59" containerName="maas-api" Oct 30 22:23:20.461891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.461892 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa24579-ce10-4820-8db8-f11a8f4e7d59" containerName="maas-api" Oct 30 22:23:20.462011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.462000 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa24579-ce10-4820-8db8-f11a8f4e7d59" containerName="maas-api" Oct 30 22:23:20.468993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.468928 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.524977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.523591 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ae8879-openshift-default-dockercfg-j62zp\"" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.579913 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.579994 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580023 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580052 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580077 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580154 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580201 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580254 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5h6s\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580331 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.586006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.580682 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm"] Oct 30 22:23:20.682317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682288 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682341 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682365 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682394 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682416 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682489 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682515 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682571 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5h6s\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682651 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.682864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.682809 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.683265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.683234 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.683434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.683289 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.683434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.683411 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.683534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.683490 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.685638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.685611 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.686263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.686239 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.746380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.746305 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.746626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.746603 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5h6s\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:20.786848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:20.786815 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:21.212647 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:23:21.212478 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf99986a_6113_49bb_bf1e_47dc26f7ddb5.slice/crio-cdc6cef39d74b64f78c42ffc5cf414e0155a37fd93ac21139702678929164219 WatchSource:0}: Error finding container cdc6cef39d74b64f78c42ffc5cf414e0155a37fd93ac21139702678929164219: Status 404 returned error can't find the container with id cdc6cef39d74b64f78c42ffc5cf414e0155a37fd93ac21139702678929164219 Oct 30 22:23:21.215664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:21.215628 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:21.215752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:21.215695 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:21.215752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:21.215725 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:21.216282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:21.216166 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm"] Oct 30 22:23:22.133583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:22.133543 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" event={"ID":"cf99986a-6113-49bb-bf1e-47dc26f7ddb5","Type":"ContainerStarted","Data":"341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4"} Oct 30 22:23:22.134039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:22.133605 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" event={"ID":"cf99986a-6113-49bb-bf1e-47dc26f7ddb5","Type":"ContainerStarted","Data":"cdc6cef39d74b64f78c42ffc5cf414e0155a37fd93ac21139702678929164219"} Oct 30 22:23:22.311499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:22.311452 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podStartSLOduration=2.3114381010000002 podStartE2EDuration="2.311438101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.308373015 +0000 UTC m=+2181.708929136" watchObservedRunningTime="2026-10-30 22:23:22.311438101 +0000 UTC m=+2181.711994221" Oct 30 22:23:22.787893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:22.787856 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:22.793881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:22.793856 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:22.867368 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:22.867333 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:23:23.140974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.140924 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:23.141895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.141878 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:23:23.506299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.506276 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:23:23.618396 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618374 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618481 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618513 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618536 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618596 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618651 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618704 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618790 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.618865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618833 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert\") pod \"9f264482-3845-4221-bbf1-95a1b3b386ae\" (UID: \"9f264482-3845-4221-bbf1-95a1b3b386ae\") " Oct 30 22:23:23.619078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618940 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:23.619078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.618988 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:23.619252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.619094 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:23.619252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.619186 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:23.619473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.619445 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:23.622353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.620095 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.622353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.620123 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.622353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.620139 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.622353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.620152 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f264482-3845-4221-bbf1-95a1b3b386ae-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.622353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.620170 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.622353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.621191 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:23.623303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.623262 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "kube-api-access-kzvfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:23.624104 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.624079 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:23.634040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.633979 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy" pod "9f264482-3845-4221-bbf1-95a1b3b386ae" (UID: "9f264482-3845-4221-bbf1-95a1b3b386ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:23.721115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.721084 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.721115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.721109 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.721115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.721119 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f264482-3845-4221-bbf1-95a1b3b386ae-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:23.721322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:23.721129 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvfg\" (UniqueName: \"kubernetes.io/projected/9f264482-3845-4221-bbf1-95a1b3b386ae-kube-api-access-kzvfg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:24.149126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.149052 5607 generic.go:356] "Generic (PLEG): container finished" podID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerID="30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1" exitCode=0 Oct 30 22:23:24.149126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.149115 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" Oct 30 22:23:24.149650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.149142 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" event={"ID":"9f264482-3845-4221-bbf1-95a1b3b386ae","Type":"ContainerDied","Data":"30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1"} Oct 30 22:23:24.149650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.149192 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw" event={"ID":"9f264482-3845-4221-bbf1-95a1b3b386ae","Type":"ContainerDied","Data":"914bbac0c89fca1b66b4a717de3a6d87b235c0e456cecbe8f12252068b730d98"} Oct 30 22:23:24.149650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.149215 5607 scope.go:122] "RemoveContainer" containerID="30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1" Oct 30 22:23:24.160750 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.160736 5607 scope.go:122] "RemoveContainer" containerID="de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078" Oct 30 22:23:24.169852 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.169837 5607 scope.go:122] "RemoveContainer" containerID="30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1" Oct 30 22:23:24.170126 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:24.170108 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1\": container with ID starting with 30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1 not found: ID does not exist" containerID="30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1" Oct 30 22:23:24.170172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.170135 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1"} err="failed to get container status \"30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1\": rpc error: code = NotFound desc = could not find container \"30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1\": container with ID starting with 30d16361ea5221705dc54165c72d615fea24fd2b758a65b8e4422e9d284356a1 not found: ID does not exist" Oct 30 22:23:24.170172 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.170149 5607 scope.go:122] "RemoveContainer" containerID="de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078" Oct 30 22:23:24.170367 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:24.170349 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078\": container with ID starting with de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078 not found: ID does not exist" containerID="de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078" Oct 30 22:23:24.170407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.170372 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078"} err="failed to get container status \"de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078\": rpc error: code = NotFound desc = could not find container \"de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078\": container with ID starting with de0bd386ed28c817aa3b16e057df20b5ab1dd93a223a15bbcc2667e9d0946078 not found: ID does not exist" Oct 30 22:23:24.369938 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.369910 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw"] Oct 30 22:23:24.421791 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.421729 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7350bc-openshift-default-5947478dcc-nh5mw"] Oct 30 22:23:24.870782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:24.870753 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" path="/var/lib/kubelet/pods/9f264482-3845-4221-bbf1-95a1b3b386ae/volumes" Oct 30 22:23:27.210582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.210552 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq"] Oct 30 22:23:27.318945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.318918 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd"] Oct 30 22:23:27.319621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.319603 5607 cpu_manager.go:414] "containerMap: removing container" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" Oct 30 22:23:27.319677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.319624 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" Oct 30 22:23:27.319677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.319652 5607 cpu_manager.go:414] "containerMap: removing container" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" Oct 30 22:23:27.319677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.319657 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" Oct 30 22:23:27.319771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.319755 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" Oct 30 22:23:27.325190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.325167 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.367543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.365219 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.367881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.367848 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.367933 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.368011 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.368075 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfdv\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.368129 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.368164 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.368201 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.368330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.368302 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.382996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.378727 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd"] Oct 30 22:23:27.388880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.386838 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc"] Oct 30 22:23:27.397477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.397447 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c89e47-openshift-default-dockercfg-99g4j\"" Oct 30 22:23:27.405236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.405200 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:23:27.469207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469109 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469163 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfdv\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469202 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469237 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469264 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469330 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469359 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469423 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469462 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.469931 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.469905 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.470161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.470143 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.470570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.470548 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.470791 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.470772 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.470939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.470918 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.474113 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.474047 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.477502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.477476 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.540998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.537749 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.541887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.541860 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfdv\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv\") pod \"e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:27.695130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.689851 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:23:27.700058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:27.700022 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:28.137350 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:23:28.137320 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcd4f8f_3f3b_46f3_b4fe_36bc4fc25fcf.slice/crio-5ed70ea812b7e96a102963912d67f1fd4ff4c3a0adcee688942d3bfae832aa0d WatchSource:0}: Error finding container 5ed70ea812b7e96a102963912d67f1fd4ff4c3a0adcee688942d3bfae832aa0d: Status 404 returned error can't find the container with id 5ed70ea812b7e96a102963912d67f1fd4ff4c3a0adcee688942d3bfae832aa0d Oct 30 22:23:28.140057 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:28.140020 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:28.140192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:28.140089 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:28.140192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:28.140121 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:28.140781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:28.140753 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd"] Oct 30 22:23:28.177635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:28.177609 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" event={"ID":"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf","Type":"ContainerStarted","Data":"5ed70ea812b7e96a102963912d67f1fd4ff4c3a0adcee688942d3bfae832aa0d"} Oct 30 22:23:29.187154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:29.187116 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" event={"ID":"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf","Type":"ContainerStarted","Data":"d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411"} Oct 30 22:23:29.313997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:29.311838 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podStartSLOduration=2.311821458 podStartE2EDuration="2.311821458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:29.31125604 +0000 UTC m=+2188.711812162" watchObservedRunningTime="2026-10-30 22:23:29.311821458 +0000 UTC m=+2188.712377582" Oct 30 22:23:29.700691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:29.700664 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:29.706494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:29.706474 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:30.195733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:30.195704 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:30.196598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:30.196577 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:23:32.211269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.211230 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" podUID="f9334a5f-3280-4adc-abb7-63c75f0af9cb" containerName="payload-processing" containerID="cri-o://4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2" gracePeriod=25 Oct 30 22:23:32.388055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.387995 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" podUID="c8ce56c8-4286-46ac-8396-02cca176bed1" containerName="payload-pre-processing" containerID="cri-o://de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73" gracePeriod=25 Oct 30 22:23:32.522670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.522640 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:23:32.662447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.662415 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g\" (UniqueName: \"kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g\") pod \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " Oct 30 22:23:32.663387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.663360 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume\") pod \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\" (UID: \"f9334a5f-3280-4adc-abb7-63c75f0af9cb\") " Oct 30 22:23:32.664574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.664550 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume" pod "f9334a5f-3280-4adc-abb7-63c75f0af9cb" (UID: "f9334a5f-3280-4adc-abb7-63c75f0af9cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:32.669539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.669509 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g" pod "f9334a5f-3280-4adc-abb7-63c75f0af9cb" (UID: "f9334a5f-3280-4adc-abb7-63c75f0af9cb"). InnerVolumeSpecName "kube-api-access-l7l7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:32.704063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.704036 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:23:32.765308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.765243 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume\") pod \"c8ce56c8-4286-46ac-8396-02cca176bed1\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " Oct 30 22:23:32.765458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.765370 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz\" (UniqueName: \"kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz\") pod \"c8ce56c8-4286-46ac-8396-02cca176bed1\" (UID: \"c8ce56c8-4286-46ac-8396-02cca176bed1\") " Oct 30 22:23:32.765586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.765565 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume" pod "c8ce56c8-4286-46ac-8396-02cca176bed1" (UID: "c8ce56c8-4286-46ac-8396-02cca176bed1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:32.765841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.765823 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7l7g\" (UniqueName: \"kubernetes.io/projected/f9334a5f-3280-4adc-abb7-63c75f0af9cb-kube-api-access-l7l7g\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.765983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.765847 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9334a5f-3280-4adc-abb7-63c75f0af9cb-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.765983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.765861 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ce56c8-4286-46ac-8396-02cca176bed1-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:32.767615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.767597 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz" pod "c8ce56c8-4286-46ac-8396-02cca176bed1" (UID: "c8ce56c8-4286-46ac-8396-02cca176bed1"). InnerVolumeSpecName "kube-api-access-96dqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:32.867665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:32.867632 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96dqz\" (UniqueName: \"kubernetes.io/projected/c8ce56c8-4286-46ac-8396-02cca176bed1-kube-api-access-96dqz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:33.215685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.215656 5607 generic.go:356] "Generic (PLEG): container finished" podID="f9334a5f-3280-4adc-abb7-63c75f0af9cb" containerID="4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2" exitCode=0 Oct 30 22:23:33.216171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.215697 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" Oct 30 22:23:33.216171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.215735 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" event={"ID":"f9334a5f-3280-4adc-abb7-63c75f0af9cb","Type":"ContainerDied","Data":"4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2"} Oct 30 22:23:33.216171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.215786 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq" event={"ID":"f9334a5f-3280-4adc-abb7-63c75f0af9cb","Type":"ContainerDied","Data":"b11c0733e931a8c8bb66b3060762ccfed32da6d64f70789ba8c237d787e043b4"} Oct 30 22:23:33.216171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.215808 5607 scope.go:122] "RemoveContainer" containerID="4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2" Oct 30 22:23:33.217191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.217164 5607 generic.go:356] "Generic (PLEG): container finished" podID="c8ce56c8-4286-46ac-8396-02cca176bed1" containerID="de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73" exitCode=0 Oct 30 22:23:33.217294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.217277 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" Oct 30 22:23:33.217382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.217315 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" event={"ID":"c8ce56c8-4286-46ac-8396-02cca176bed1","Type":"ContainerDied","Data":"de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73"} Oct 30 22:23:33.217382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.217354 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc" event={"ID":"c8ce56c8-4286-46ac-8396-02cca176bed1","Type":"ContainerDied","Data":"e58552bbe3c949f1ebe22f3edc2d1d38959bc4d4073b2ff2bc85b5f26ae7fbb9"} Oct 30 22:23:33.240326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.240307 5607 scope.go:122] "RemoveContainer" containerID="4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2" Oct 30 22:23:33.240587 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:33.240568 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2\": container with ID starting with 4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2 not found: ID does not exist" containerID="4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2" Oct 30 22:23:33.240633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.240597 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2"} err="failed to get container status \"4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2\": rpc error: code = NotFound desc = could not find container \"4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2\": container with ID starting with 4b607ab00d7f5e75ad9d7b5cf56d428f3ca94d5eab942f39faf0ea7bdc1e3ca2 not found: ID does not exist" Oct 30 22:23:33.240633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.240621 5607 scope.go:122] "RemoveContainer" containerID="de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73" Oct 30 22:23:33.249847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.249829 5607 scope.go:122] "RemoveContainer" containerID="de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73" Oct 30 22:23:33.250133 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:33.250113 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73\": container with ID starting with de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73 not found: ID does not exist" containerID="de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73" Oct 30 22:23:33.250252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.250141 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73"} err="failed to get container status \"de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73\": rpc error: code = NotFound desc = could not find container \"de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73\": container with ID starting with de7a84023d6a37eb0e645ee31a1304eacfbff66ce734061422978ec0865b2e73 not found: ID does not exist" Oct 30 22:23:33.387871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.387845 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq"] Oct 30 22:23:33.438613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.438586 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-872f4a-69db496db9-9xcfq"] Oct 30 22:23:33.554637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.554580 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc"] Oct 30 22:23:33.559286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:33.559264 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-872f4a-b849bfd67-4nwrc"] Oct 30 22:23:34.871007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:34.870975 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ce56c8-4286-46ac-8396-02cca176bed1" path="/var/lib/kubelet/pods/c8ce56c8-4286-46ac-8396-02cca176bed1/volumes" Oct 30 22:23:34.871404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:34.871382 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9334a5f-3280-4adc-abb7-63c75f0af9cb" path="/var/lib/kubelet/pods/f9334a5f-3280-4adc-abb7-63c75f0af9cb/volumes" Oct 30 22:23:39.386561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.386526 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t"] Oct 30 22:23:39.387220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387160 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c8ce56c8-4286-46ac-8396-02cca176bed1" containerName="payload-pre-processing" Oct 30 22:23:39.387220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387176 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ce56c8-4286-46ac-8396-02cca176bed1" containerName="payload-pre-processing" Oct 30 22:23:39.387220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387218 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f9334a5f-3280-4adc-abb7-63c75f0af9cb" containerName="payload-processing" Oct 30 22:23:39.387220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387225 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9334a5f-3280-4adc-abb7-63c75f0af9cb" containerName="payload-processing" Oct 30 22:23:39.387516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387334 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f264482-3845-4221-bbf1-95a1b3b386ae" containerName="istio-proxy" Oct 30 22:23:39.387516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387347 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ce56c8-4286-46ac-8396-02cca176bed1" containerName="payload-pre-processing" Oct 30 22:23:39.387516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.387359 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9334a5f-3280-4adc-abb7-63c75f0af9cb" containerName="payload-processing" Oct 30 22:23:39.392337 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.392316 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.449271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.449241 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t"] Oct 30 22:23:39.499326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.499293 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c89e47-dockercfg-924p6\"" Oct 30 22:23:39.499667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.499611 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c89e47\"" Oct 30 22:23:39.550586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.550543 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.550794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.550639 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbvv\" (UniqueName: \"kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv\") pod \"payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.553040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.553010 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl"] Oct 30 22:23:39.558501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.558479 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.580878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.580854 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl"] Oct 30 22:23:39.651512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.651446 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcrm\" (UniqueName: \"kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm\") pod \"payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.651512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.651487 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.651668 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.651554 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.651668 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.651629 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbvv\" (UniqueName: \"kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv\") pod \"payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.652204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.652187 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.714665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.714635 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbvv\" (UniqueName: \"kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv\") pod \"payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:39.753036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.753008 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcrm\" (UniqueName: \"kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm\") pod \"payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.753157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.753051 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.753723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.753700 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.815746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.815723 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcrm\" (UniqueName: \"kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm\") pod \"payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:39.874527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:39.874508 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:40.006394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:40.006358 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:40.252181 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:23:40.252147 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42cfd2b_0420_4e98_b778_6f0c80ec1a4a.slice/crio-9649049f9dcf33307106105ac4c75c1e43a1d06374d28165132e2ddf2d3d3ec3 WatchSource:0}: Error finding container 9649049f9dcf33307106105ac4c75c1e43a1d06374d28165132e2ddf2d3d3ec3: Status 404 returned error can't find the container with id 9649049f9dcf33307106105ac4c75c1e43a1d06374d28165132e2ddf2d3d3ec3 Oct 30 22:23:40.259876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:40.259841 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl"] Oct 30 22:23:40.285316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:40.285282 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" event={"ID":"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a","Type":"ContainerStarted","Data":"9649049f9dcf33307106105ac4c75c1e43a1d06374d28165132e2ddf2d3d3ec3"} Oct 30 22:23:40.384475 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:23:40.384443 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8b05ae_b30f_472d_9658_912a7ec57a64.slice/crio-dd3e7b593328f0dbc39911c6d981e5d629b4c29dcb94f0f30c2cbd9b4f3fd006 WatchSource:0}: Error finding container dd3e7b593328f0dbc39911c6d981e5d629b4c29dcb94f0f30c2cbd9b4f3fd006: Status 404 returned error can't find the container with id dd3e7b593328f0dbc39911c6d981e5d629b4c29dcb94f0f30c2cbd9b4f3fd006 Oct 30 22:23:40.388523 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:40.388499 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t"] Oct 30 22:23:41.293390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.293351 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" event={"ID":"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a","Type":"ContainerStarted","Data":"483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878"} Oct 30 22:23:41.293572 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.293451 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:41.294929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.294904 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" event={"ID":"6c8b05ae-b30f-472d-9658-912a7ec57a64","Type":"ContainerStarted","Data":"f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777"} Oct 30 22:23:41.295071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.294941 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" event={"ID":"6c8b05ae-b30f-472d-9658-912a7ec57a64","Type":"ContainerStarted","Data":"dd3e7b593328f0dbc39911c6d981e5d629b4c29dcb94f0f30c2cbd9b4f3fd006"} Oct 30 22:23:41.295071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.295048 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:41.415716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.415668 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" podStartSLOduration=2.415657596 podStartE2EDuration="2.415657596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:41.413937779 +0000 UTC m=+2200.814493901" watchObservedRunningTime="2026-10-30 22:23:41.415657596 +0000 UTC m=+2200.816213775" Oct 30 22:23:41.536397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:41.534385 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" podStartSLOduration=2.5343657349999997 podStartE2EDuration="2.534365735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:41.533025013 +0000 UTC m=+2200.933581136" watchObservedRunningTime="2026-10-30 22:23:41.534365735 +0000 UTC m=+2200.934921857" Oct 30 22:23:52.306372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:52.306341 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:23:52.306873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:23:52.306736 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:23:52.864558 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:23:52.864521 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:24:04.730821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.730785 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm"] Oct 30 22:24:04.736520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.736501 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.792116 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.792084 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-923d78-serving-cert\"" Oct 30 22:24:04.834687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.834655 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882hx\" (UniqueName: \"kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.834833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.834808 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.834905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.834867 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.835040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.835024 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.845976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.845927 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm"] Oct 30 22:24:04.936467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.936441 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.936606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.936518 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.936606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.936590 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882hx\" (UniqueName: \"kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.936680 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:24:04.936656 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-serving-cert: secret "maas-api-e2e-praxis-b-923d78-serving-cert" not found Oct 30 22:24:04.936721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.936703 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.936763 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:24:04.936724 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls podName:f355f307-0c39-4050-8cec-ba3f56a0234a nodeName:}" failed. No retries permitted until 2026-10-30 22:24:05.436703402 +0000 UTC m=+2224.837259526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls") pod "maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" (UID: "f355f307-0c39-4050-8cec-ba3f56a0234a") : secret "maas-api-e2e-praxis-b-923d78-serving-cert" not found Oct 30 22:24:04.937344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.937274 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:04.939119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:04.939097 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:05.004794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:05.004723 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-882hx\" (UniqueName: \"kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:05.444276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:05.444244 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:05.446833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:05.446810 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:05.650827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:05.650799 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:06.024657 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:24:06.024608 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf355f307_0c39_4050_8cec_ba3f56a0234a.slice/crio-e4b0a257553f558f7826f0dfe22e1dfa8ccfb6a521de3c5e0a2f5a940d723ed6 WatchSource:0}: Error finding container e4b0a257553f558f7826f0dfe22e1dfa8ccfb6a521de3c5e0a2f5a940d723ed6: Status 404 returned error can't find the container with id e4b0a257553f558f7826f0dfe22e1dfa8ccfb6a521de3c5e0a2f5a940d723ed6 Oct 30 22:24:06.029103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:06.029074 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm"] Oct 30 22:24:06.461138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:06.461101 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" event={"ID":"f355f307-0c39-4050-8cec-ba3f56a0234a","Type":"ContainerStarted","Data":"e4b0a257553f558f7826f0dfe22e1dfa8ccfb6a521de3c5e0a2f5a940d723ed6"} Oct 30 22:24:07.468945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:07.468913 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" event={"ID":"f355f307-0c39-4050-8cec-ba3f56a0234a","Type":"ContainerStarted","Data":"318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f"} Oct 30 22:24:07.469352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:07.469053 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:07.589697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:07.589650 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" podStartSLOduration=2.678688417 podStartE2EDuration="3.589634359s" totalImagesPullingTime="910.945942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:07.588654441 +0000 UTC m=+2226.989210562" watchObservedRunningTime="2026-10-30 22:24:07.589634359 +0000 UTC m=+2226.990190484" Oct 30 22:24:13.481585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:13.481557 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:24:22.864208 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:24:22.864169 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:24:31.487315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.487281 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s"] Oct 30 22:24:31.491676 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.491659 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.550773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.550738 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ae8879-serving-cert\"" Oct 30 22:24:31.593058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.593001 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.593317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.593060 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.593317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.593120 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.593536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.593382 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp4df\" (UniqueName: \"kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.605020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.604992 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s"] Oct 30 22:24:31.695115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.695078 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.695115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.695121 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.695363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.695162 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.695363 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:24:31.695221 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-serving-cert: secret "maas-api-e2e-praxis-a-ae8879-serving-cert" not found Oct 30 22:24:31.695363 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:24:31.695290 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls podName:fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae nodeName:}" failed. No retries permitted until 2026-10-30 22:24:32.195273117 +0000 UTC m=+2251.595829217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls") pod "maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" (UID: "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae") : secret "maas-api-e2e-praxis-a-ae8879-serving-cert" not found Oct 30 22:24:31.695519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.695432 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp4df\" (UniqueName: \"kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.695899 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.695874 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.697915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.697895 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:31.759827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:31.759751 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp4df\" (UniqueName: \"kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:32.205125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:32.205088 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:32.207827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:32.207794 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:32.408719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:32.408675 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:32.806815 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:24:32.806781 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa55ffc1_b2a9_43ce_93cc_43bfe80d6aae.slice/crio-9df69d07ec5cbe7d271579edcaeedce070e6bb60c56a31e6e58fc345b8e847d1 WatchSource:0}: Error finding container 9df69d07ec5cbe7d271579edcaeedce070e6bb60c56a31e6e58fc345b8e847d1: Status 404 returned error can't find the container with id 9df69d07ec5cbe7d271579edcaeedce070e6bb60c56a31e6e58fc345b8e847d1 Oct 30 22:24:32.811285 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:32.811258 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s"] Oct 30 22:24:33.670140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:33.670096 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" event={"ID":"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae","Type":"ContainerStarted","Data":"9df69d07ec5cbe7d271579edcaeedce070e6bb60c56a31e6e58fc345b8e847d1"} Oct 30 22:24:34.678717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.678642 5607 generic.go:356] "Generic (PLEG): container finished" podID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerID="10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21" exitCode=0 Oct 30 22:24:34.679258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.678719 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" event={"ID":"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d","Type":"ContainerDied","Data":"10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21"} Oct 30 22:24:34.679258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.679165 5607 scope.go:122] "RemoveContainer" containerID="10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21" Oct 30 22:24:34.680702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.680675 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" event={"ID":"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae","Type":"ContainerStarted","Data":"3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e"} Oct 30 22:24:34.680842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.680730 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:34.680907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.680859 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:34.681007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.680986 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:34.681080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.681050 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:34.973204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:34.973152 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" podStartSLOduration=3.043944532 podStartE2EDuration="3.973138481s" totalImagesPullingTime="929.193949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:34.971946813 +0000 UTC m=+2254.372502934" watchObservedRunningTime="2026-10-30 22:24:34.973138481 +0000 UTC m=+2254.373694602" Oct 30 22:24:35.689732 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:35.689696 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" event={"ID":"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d","Type":"ContainerStarted","Data":"9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60"} Oct 30 22:24:36.573503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:36.573467 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:24:36.573742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:36.573722 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:24:36.573800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:36.573768 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:24:37.573127 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:37.573086 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:24:37.573560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:37.573136 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:24:38.572857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:38.572823 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:24:38.573067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:38.572885 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:24:39.573527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:39.573495 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:24:39.573979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:39.573543 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:24:40.573440 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:40.573362 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:24:40.573595 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:40.573441 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:24:40.695757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:40.695728 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:24:41.577664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:41.577629 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:24:41.578187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:41.578171 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:24:41.578906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:41.578885 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:24:52.864654 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:24:52.864620 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:24:53.658031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:53.658000 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:24:53.918231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:53.918160 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:24:54.183170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.183091 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:24:54.183170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.183157 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:54.891028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.890925 5607 generic.go:356] "Generic (PLEG): container finished" podID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerID="341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4" exitCode=0 Oct 30 22:24:54.891028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.890994 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" event={"ID":"cf99986a-6113-49bb-bf1e-47dc26f7ddb5","Type":"ContainerDied","Data":"341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4"} Oct 30 22:24:54.891410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.891391 5607 scope.go:122] "RemoveContainer" containerID="341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4" Oct 30 22:24:54.893070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.893032 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:54.893156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.893113 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:54.893156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:54.893147 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:24:55.905981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:55.901251 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" event={"ID":"cf99986a-6113-49bb-bf1e-47dc26f7ddb5","Type":"ContainerStarted","Data":"a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a"} Oct 30 22:24:56.787233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:56.787199 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:24:56.787471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:56.787392 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 22:24:56.787554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:56.787464 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 22:24:57.787501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:57.787461 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 22:24:57.787907 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:57.787509 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 22:24:58.787979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:58.787924 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 22:24:58.788383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:58.788012 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 22:24:59.788251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:59.788217 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 22:24:59.788658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:24:59.788278 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 22:25:00.788210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:00.788179 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 22:25:00.788374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:00.788228 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 22:25:01.791533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:01.791504 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:25:01.792108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:01.791983 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:25:01.792883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:01.792865 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:25:21.747784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:21.747746 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8"] Oct 30 22:25:21.860219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:21.860175 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk"] Oct 30 22:25:22.864298 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:25:22.864263 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:25:25.121418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:25.121383 5607 generic.go:356] "Generic (PLEG): container finished" podID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerID="d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411" exitCode=0 Oct 30 22:25:25.121886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:25.121448 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" event={"ID":"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf","Type":"ContainerDied","Data":"d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411"} Oct 30 22:25:25.121886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:25.121770 5607 scope.go:122] "RemoveContainer" containerID="d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411" Oct 30 22:25:25.123360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:25.123325 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:25.123475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:25.123407 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:25.123475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:25.123441 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:25:26.130457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:26.130420 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" event={"ID":"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf","Type":"ContainerStarted","Data":"4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9"} Oct 30 22:25:26.714977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:26.710363 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:25:26.724610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:26.723126 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 22:25:26.724610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:26.723193 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 22:25:26.749141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:26.749081 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" podUID="684d0c5a-a014-4672-95be-b2538a7ccd1d" containerName="payload-processing" containerID="cri-o://7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc" gracePeriod=25 Oct 30 22:25:26.860715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:26.860673 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" podUID="5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" containerName="payload-pre-processing" containerID="cri-o://74a65210761dc8abffd65d814a8f5a8cd301bdb8368ff076562e46ba180c2fd2" gracePeriod=25 Oct 30 22:25:27.077244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.077207 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:25:27.134421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.134384 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume\") pod \"684d0c5a-a014-4672-95be-b2538a7ccd1d\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " Oct 30 22:25:27.134917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.134581 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp\" (UniqueName: \"kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp\") pod \"684d0c5a-a014-4672-95be-b2538a7ccd1d\" (UID: \"684d0c5a-a014-4672-95be-b2538a7ccd1d\") " Oct 30 22:25:27.135972 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.135908 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume" pod "684d0c5a-a014-4672-95be-b2538a7ccd1d" (UID: "684d0c5a-a014-4672-95be-b2538a7ccd1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:27.138506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.138475 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp" pod "684d0c5a-a014-4672-95be-b2538a7ccd1d" (UID: "684d0c5a-a014-4672-95be-b2538a7ccd1d"). InnerVolumeSpecName "kube-api-access-29vgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:27.146715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.146682 5607 generic.go:356] "Generic (PLEG): container finished" podID="684d0c5a-a014-4672-95be-b2538a7ccd1d" containerID="7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc" exitCode=0 Oct 30 22:25:27.146819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.146771 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" event={"ID":"684d0c5a-a014-4672-95be-b2538a7ccd1d","Type":"ContainerDied","Data":"7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc"} Oct 30 22:25:27.146866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.146820 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" event={"ID":"684d0c5a-a014-4672-95be-b2538a7ccd1d","Type":"ContainerDied","Data":"d1fc3ea90c4e1fd689d1d34716c8c04973eedba69f7f71cb583dec287ace0a06"} Oct 30 22:25:27.146866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.146852 5607 scope.go:122] "RemoveContainer" containerID="7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc" Oct 30 22:25:27.147114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.147097 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8" Oct 30 22:25:27.152083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.152062 5607 generic.go:356] "Generic (PLEG): container finished" podID="5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" containerID="74a65210761dc8abffd65d814a8f5a8cd301bdb8368ff076562e46ba180c2fd2" exitCode=0 Oct 30 22:25:27.152776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.152752 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" event={"ID":"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f","Type":"ContainerDied","Data":"74a65210761dc8abffd65d814a8f5a8cd301bdb8368ff076562e46ba180c2fd2"} Oct 30 22:25:27.168677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.168657 5607 scope.go:122] "RemoveContainer" containerID="7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc" Oct 30 22:25:27.168976 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:25:27.168939 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc\": container with ID starting with 7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc not found: ID does not exist" containerID="7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc" Oct 30 22:25:27.169066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.168986 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc"} err="failed to get container status \"7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc\": rpc error: code = NotFound desc = could not find container \"7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc\": container with ID starting with 7853bfa2628ff05fba5941658b4c05bfd4af74a114956d21fa80e7a0f7f77ffc not found: ID does not exist" Oct 30 22:25:27.178648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.178625 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:25:27.236627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.236551 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw\" (UniqueName: \"kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw\") pod \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " Oct 30 22:25:27.236795 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.236769 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume\") pod \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\" (UID: \"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f\") " Oct 30 22:25:27.237369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.237205 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume" pod "5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" (UID: "5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:27.237519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.237498 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:27.237581 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.237527 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vgp\" (UniqueName: \"kubernetes.io/projected/684d0c5a-a014-4672-95be-b2538a7ccd1d-kube-api-access-29vgp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:27.237581 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.237543 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/684d0c5a-a014-4672-95be-b2538a7ccd1d-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:27.240081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.240055 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw" pod "5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" (UID: "5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f"). InnerVolumeSpecName "kube-api-access-q25tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:27.337086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.337055 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8"] Oct 30 22:25:27.338689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.338668 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q25tw\" (UniqueName: \"kubernetes.io/projected/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f-kube-api-access-q25tw\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:27.346414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.346387 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bd7fe-8468ff4697-wlwx8"] Oct 30 22:25:27.710981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.702266 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 22:25:27.710981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:27.702322 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 22:25:28.161763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.161677 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" event={"ID":"5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f","Type":"ContainerDied","Data":"a53fafa7fa7a296b42caba26ec517d17744283b95b8c9b2d4b251dcc814d4afc"} Oct 30 22:25:28.161763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.161742 5607 scope.go:122] "RemoveContainer" containerID="74a65210761dc8abffd65d814a8f5a8cd301bdb8368ff076562e46ba180c2fd2" Oct 30 22:25:28.162351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.161691 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk" Oct 30 22:25:28.335336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.335299 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk"] Oct 30 22:25:28.340026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.340000 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bd7fe-68d9f4cc99-tksgk"] Oct 30 22:25:28.717979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.714371 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 22:25:28.717979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.714433 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 22:25:28.872081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.872050 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" path="/var/lib/kubelet/pods/5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f/volumes" Oct 30 22:25:28.872482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:28.872469 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684d0c5a-a014-4672-95be-b2538a7ccd1d" path="/var/lib/kubelet/pods/684d0c5a-a014-4672-95be-b2538a7ccd1d/volumes" Oct 30 22:25:29.707979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:29.707369 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 22:25:29.707979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:29.707429 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 22:25:30.214978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:30.211082 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:25:30.713977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:30.713305 5607 patch_prober.go:29] interesting pod/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 22:25:30.713977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:30.713371 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 22:25:31.706239 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:31.706209 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:25:31.707331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:31.707305 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:25:34.498882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:34.498847 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w"] Oct 30 22:25:34.614671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:34.614629 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss"] Oct 30 22:25:39.499919 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.499863 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" podUID="c3e471e0-0951-4192-86d4-dce93dfa8be1" containerName="payload-processing" containerID="cri-o://b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58" gracePeriod=25 Oct 30 22:25:39.615292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.615254 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" podUID="c0a9fa25-6aaa-4416-a247-e1e92edd51e3" containerName="payload-pre-processing" containerID="cri-o://a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60" gracePeriod=25 Oct 30 22:25:39.769997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.769908 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:25:39.887110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.887083 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:25:39.935586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.935549 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw\" (UniqueName: \"kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw\") pod \"c3e471e0-0951-4192-86d4-dce93dfa8be1\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " Oct 30 22:25:39.935886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.935864 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume\") pod \"c3e471e0-0951-4192-86d4-dce93dfa8be1\" (UID: \"c3e471e0-0951-4192-86d4-dce93dfa8be1\") " Oct 30 22:25:39.936258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.936234 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume" pod "c3e471e0-0951-4192-86d4-dce93dfa8be1" (UID: "c3e471e0-0951-4192-86d4-dce93dfa8be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:39.936480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.936459 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e471e0-0951-4192-86d4-dce93dfa8be1-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:39.937763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:39.937741 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw" pod "c3e471e0-0951-4192-86d4-dce93dfa8be1" (UID: "c3e471e0-0951-4192-86d4-dce93dfa8be1"). InnerVolumeSpecName "kube-api-access-plvkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:40.037493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.037404 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume\") pod \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " Oct 30 22:25:40.037493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.037495 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb\" (UniqueName: \"kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb\") pod \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\" (UID: \"c0a9fa25-6aaa-4416-a247-e1e92edd51e3\") " Oct 30 22:25:40.037826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.037802 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume" pod "c0a9fa25-6aaa-4416-a247-e1e92edd51e3" (UID: "c0a9fa25-6aaa-4416-a247-e1e92edd51e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:40.037921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.037856 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvkw\" (UniqueName: \"kubernetes.io/projected/c3e471e0-0951-4192-86d4-dce93dfa8be1-kube-api-access-plvkw\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.039853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.039826 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb" pod "c0a9fa25-6aaa-4416-a247-e1e92edd51e3" (UID: "c0a9fa25-6aaa-4416-a247-e1e92edd51e3"). InnerVolumeSpecName "kube-api-access-sqbxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:40.139361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.139319 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbxb\" (UniqueName: \"kubernetes.io/projected/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-kube-api-access-sqbxb\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.139361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.139356 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0a9fa25-6aaa-4416-a247-e1e92edd51e3-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.257118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.257074 5607 generic.go:356] "Generic (PLEG): container finished" podID="c0a9fa25-6aaa-4416-a247-e1e92edd51e3" containerID="a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60" exitCode=0 Oct 30 22:25:40.257318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.257125 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" Oct 30 22:25:40.257318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.257161 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" event={"ID":"c0a9fa25-6aaa-4416-a247-e1e92edd51e3","Type":"ContainerDied","Data":"a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60"} Oct 30 22:25:40.257318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.257230 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss" event={"ID":"c0a9fa25-6aaa-4416-a247-e1e92edd51e3","Type":"ContainerDied","Data":"901af9e3f5aceed3051dd2b8bed1696cba04bda87f96af741ca42ba13cbb888c"} Oct 30 22:25:40.257318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.257257 5607 scope.go:122] "RemoveContainer" containerID="a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60" Oct 30 22:25:40.258698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.258587 5607 generic.go:356] "Generic (PLEG): container finished" podID="c3e471e0-0951-4192-86d4-dce93dfa8be1" containerID="b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58" exitCode=0 Oct 30 22:25:40.258698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.258615 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" event={"ID":"c3e471e0-0951-4192-86d4-dce93dfa8be1","Type":"ContainerDied","Data":"b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58"} Oct 30 22:25:40.258698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.258620 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" Oct 30 22:25:40.258698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.258649 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w" event={"ID":"c3e471e0-0951-4192-86d4-dce93dfa8be1","Type":"ContainerDied","Data":"01c55cf4e66cd7fa5867b55b6ec35c304e01703fddcdb4aa4da6a66838e74173"} Oct 30 22:25:40.269082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.269063 5607 scope.go:122] "RemoveContainer" containerID="a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60" Oct 30 22:25:40.269375 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:25:40.269347 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60\": container with ID starting with a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60 not found: ID does not exist" containerID="a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60" Oct 30 22:25:40.269495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.269386 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60"} err="failed to get container status \"a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60\": rpc error: code = NotFound desc = could not find container \"a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60\": container with ID starting with a7c7185970572add30308e56b13b2c2cb4f12b76fd25e13c01565cf43ba8dc60 not found: ID does not exist" Oct 30 22:25:40.269495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.269405 5607 scope.go:122] "RemoveContainer" containerID="b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58" Oct 30 22:25:40.278774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.278756 5607 scope.go:122] "RemoveContainer" containerID="b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58" Oct 30 22:25:40.279057 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:25:40.279037 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58\": container with ID starting with b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58 not found: ID does not exist" containerID="b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58" Oct 30 22:25:40.279115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.279063 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58"} err="failed to get container status \"b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58\": rpc error: code = NotFound desc = could not find container \"b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58\": container with ID starting with b1395aa3ae8bf39be8f2844a843e29113430e8c62781d952ad869b62dd028f58 not found: ID does not exist" Oct 30 22:25:40.426032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.425942 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss"] Oct 30 22:25:40.429585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.429555 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-40fb42-677f5fcd55-rfdss"] Oct 30 22:25:40.542089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.542062 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w"] Oct 30 22:25:40.547434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.547414 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-40fb42-77fcf46b47-tj85w"] Oct 30 22:25:40.872534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.872504 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a9fa25-6aaa-4416-a247-e1e92edd51e3" path="/var/lib/kubelet/pods/c0a9fa25-6aaa-4416-a247-e1e92edd51e3/volumes" Oct 30 22:25:40.872921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:40.872907 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e471e0-0951-4192-86d4-dce93dfa8be1" path="/var/lib/kubelet/pods/c3e471e0-0951-4192-86d4-dce93dfa8be1/volumes" Oct 30 22:25:52.863440 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:25:52.863402 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:25:58.363259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363225 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6"] Oct 30 22:25:58.363874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363860 5607 cpu_manager.go:414] "containerMap: removing container" podUID="5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" containerName="payload-pre-processing" Oct 30 22:25:58.363922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363875 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" containerName="payload-pre-processing" Oct 30 22:25:58.363922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363903 5607 cpu_manager.go:414] "containerMap: removing container" podUID="684d0c5a-a014-4672-95be-b2538a7ccd1d" containerName="payload-processing" Oct 30 22:25:58.363922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363908 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684d0c5a-a014-4672-95be-b2538a7ccd1d" containerName="payload-processing" Oct 30 22:25:58.364066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363928 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c3e471e0-0951-4192-86d4-dce93dfa8be1" containerName="payload-processing" Oct 30 22:25:58.364066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363932 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e471e0-0951-4192-86d4-dce93dfa8be1" containerName="payload-processing" Oct 30 22:25:58.364066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363940 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c0a9fa25-6aaa-4416-a247-e1e92edd51e3" containerName="payload-pre-processing" Oct 30 22:25:58.364066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.363945 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a9fa25-6aaa-4416-a247-e1e92edd51e3" containerName="payload-pre-processing" Oct 30 22:25:58.364189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.364085 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="684d0c5a-a014-4672-95be-b2538a7ccd1d" containerName="payload-processing" Oct 30 22:25:58.364189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.364097 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e471e0-0951-4192-86d4-dce93dfa8be1" containerName="payload-processing" Oct 30 22:25:58.364189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.364102 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a9fa25-6aaa-4416-a247-e1e92edd51e3" containerName="payload-pre-processing" Oct 30 22:25:58.364189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.364112 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2a7cc6-5af5-4efc-b529-1cfd5c52da0f" containerName="payload-pre-processing" Oct 30 22:25:58.366394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.366378 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.405744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.405717 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpmq\" (UniqueName: \"kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq\") pod \"payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.405969 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.405939 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.423342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.423319 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cf5019\"" Oct 30 22:25:58.474353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.474330 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cf5019-dockercfg-5xbt4\"" Oct 30 22:25:58.477260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.477241 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6"] Oct 30 22:25:58.507313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.507289 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpmq\" (UniqueName: \"kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq\") pod \"payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.507493 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.507470 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.508151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.508134 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.532046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.532023 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg"] Oct 30 22:25:58.535161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.535143 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.594982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.594865 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg"] Oct 30 22:25:58.647028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.646834 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlddj\" (UniqueName: \"kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj\") pod \"payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.647028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.647007 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.648070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.648039 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpmq\" (UniqueName: \"kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq\") pod \"payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.679182 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.679151 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:58.755066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.753109 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlddj\" (UniqueName: \"kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj\") pod \"payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.755066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.753172 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.755066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.754191 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.815773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.815744 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlddj\" (UniqueName: \"kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj\") pod \"payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:58.847642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:58.847617 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:59.056940 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:25:59.056914 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97262a5f_97ba_4fc8_8ee0_02401e8d76ec.slice/crio-dc1e848148e60e49dc256ecbb68cdd4e789c413a2136095518d78ac566850e1c WatchSource:0}: Error finding container dc1e848148e60e49dc256ecbb68cdd4e789c413a2136095518d78ac566850e1c: Status 404 returned error can't find the container with id dc1e848148e60e49dc256ecbb68cdd4e789c413a2136095518d78ac566850e1c Oct 30 22:25:59.060354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.060331 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6"] Oct 30 22:25:59.218699 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:25:59.218674 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fc2fef_6c84_437b_a09d_4b6da1170763.slice/crio-a4fde8d78013f308bc883a0055cd747301cb18142f8e374778c8348dc3c32bde WatchSource:0}: Error finding container a4fde8d78013f308bc883a0055cd747301cb18142f8e374778c8348dc3c32bde: Status 404 returned error can't find the container with id a4fde8d78013f308bc883a0055cd747301cb18142f8e374778c8348dc3c32bde Oct 30 22:25:59.222364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.222341 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg"] Oct 30 22:25:59.405682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.405636 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" event={"ID":"78fc2fef-6c84-437b-a09d-4b6da1170763","Type":"ContainerStarted","Data":"005879c80aba8b3973542502e3a810f6090c7a55145312c516d04819378571b5"} Oct 30 22:25:59.406161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.405707 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:25:59.406161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.405730 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" event={"ID":"78fc2fef-6c84-437b-a09d-4b6da1170763","Type":"ContainerStarted","Data":"a4fde8d78013f308bc883a0055cd747301cb18142f8e374778c8348dc3c32bde"} Oct 30 22:25:59.407530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.407504 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" event={"ID":"97262a5f-97ba-4fc8-8ee0-02401e8d76ec","Type":"ContainerStarted","Data":"0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c"} Oct 30 22:25:59.407645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.407547 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" event={"ID":"97262a5f-97ba-4fc8-8ee0-02401e8d76ec","Type":"ContainerStarted","Data":"dc1e848148e60e49dc256ecbb68cdd4e789c413a2136095518d78ac566850e1c"} Oct 30 22:25:59.407645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.407637 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:25:59.526854 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.526757 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" podStartSLOduration=1.526740174 podStartE2EDuration="1.526740174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:59.523272716 +0000 UTC m=+2338.923828838" watchObservedRunningTime="2026-10-30 22:25:59.526740174 +0000 UTC m=+2338.927296299" Oct 30 22:25:59.644298 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:25:59.644217 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" podStartSLOduration=1.644197313 podStartE2EDuration="1.644197313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:59.641039788 +0000 UTC m=+2339.041595911" watchObservedRunningTime="2026-10-30 22:25:59.644197313 +0000 UTC m=+2339.044753433" Oct 30 22:26:07.808330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:07.808292 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl"] Oct 30 22:26:07.923376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:07.923345 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t"] Oct 30 22:26:10.417712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:10.417637 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:26:10.417712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:10.417700 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:26:12.186810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.186775 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg"] Oct 30 22:26:12.187272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.187021 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" podUID="2740c3d2-b33f-4e20-9deb-6538a4e9413f" containerName="maas-api" containerID="cri-o://9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a" gracePeriod=30 Oct 30 22:26:12.476359 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.476336 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:26:12.496128 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.496098 5607 generic.go:356] "Generic (PLEG): container finished" podID="2740c3d2-b33f-4e20-9deb-6538a4e9413f" containerID="9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a" exitCode=0 Oct 30 22:26:12.496269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.496140 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" event={"ID":"2740c3d2-b33f-4e20-9deb-6538a4e9413f","Type":"ContainerDied","Data":"9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a"} Oct 30 22:26:12.496269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.496208 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" event={"ID":"2740c3d2-b33f-4e20-9deb-6538a4e9413f","Type":"ContainerDied","Data":"3294ba272fdfffcd4117bcdbf15997cc3d59a2aea70a2e5453d2162cf1598d78"} Oct 30 22:26:12.496269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.496156 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg" Oct 30 22:26:12.496269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.496240 5607 scope.go:122] "RemoveContainer" containerID="9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a" Oct 30 22:26:12.506149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.506132 5607 scope.go:122] "RemoveContainer" containerID="9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a" Oct 30 22:26:12.506395 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:12.506377 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a\": container with ID starting with 9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a not found: ID does not exist" containerID="9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a" Oct 30 22:26:12.506443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.506403 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a"} err="failed to get container status \"9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a\": rpc error: code = NotFound desc = could not find container \"9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a\": container with ID starting with 9358758d79ca64a9d2fe5cab70b6d40825f6ac6c51012a4d7967e2705b6c239a not found: ID does not exist" Oct 30 22:26:12.630969 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.630926 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca\") pod \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " Oct 30 22:26:12.631166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.631049 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\") pod \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " Oct 30 22:26:12.631166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.631141 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs\") pod \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " Oct 30 22:26:12.631302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.631192 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq\" (UniqueName: \"kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq\") pod \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\" (UID: \"2740c3d2-b33f-4e20-9deb-6538a4e9413f\") " Oct 30 22:26:12.631368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.631341 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca" pod "2740c3d2-b33f-4e20-9deb-6538a4e9413f" (UID: "2740c3d2-b33f-4e20-9deb-6538a4e9413f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:12.631923 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.631831 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2740c3d2-b33f-4e20-9deb-6538a4e9413f-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:12.633759 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.633734 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs" pod "2740c3d2-b33f-4e20-9deb-6538a4e9413f" (UID: "2740c3d2-b33f-4e20-9deb-6538a4e9413f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:12.634194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.634174 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq" pod "2740c3d2-b33f-4e20-9deb-6538a4e9413f" (UID: "2740c3d2-b33f-4e20-9deb-6538a4e9413f"). InnerVolumeSpecName "kube-api-access-k88wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:12.634194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.634170 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls" pod "2740c3d2-b33f-4e20-9deb-6538a4e9413f" (UID: "2740c3d2-b33f-4e20-9deb-6538a4e9413f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:12.733619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.733527 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:12.733619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.733568 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2740c3d2-b33f-4e20-9deb-6538a4e9413f-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:12.733619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.733584 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88wq\" (UniqueName: \"kubernetes.io/projected/2740c3d2-b33f-4e20-9deb-6538a4e9413f-kube-api-access-k88wq\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:12.808564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.808532 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" podUID="a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" containerName="payload-processing" containerID="cri-o://483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878" gracePeriod=25 Oct 30 22:26:12.923758 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.923729 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" podUID="6c8b05ae-b30f-472d-9658-912a7ec57a64" containerName="payload-pre-processing" containerID="cri-o://f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777" gracePeriod=25 Oct 30 22:26:12.933417 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.933389 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg"] Oct 30 22:26:12.949190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:12.949163 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-40fb42-7b4d68644b-cwvtg"] Oct 30 22:26:13.153735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.147595 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:26:13.239939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.239888 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm\" (UniqueName: \"kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm\") pod \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " Oct 30 22:26:13.240476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.240022 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume\") pod \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\" (UID: \"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a\") " Oct 30 22:26:13.240476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.240424 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume" pod "a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" (UID: "a42cfd2b-0420-4e98-b778-6f0c80ec1a4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:13.240666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.240647 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.246184 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.246127 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm" pod "a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" (UID: "a42cfd2b-0420-4e98-b778-6f0c80ec1a4a"). InnerVolumeSpecName "kube-api-access-mjcrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:13.271554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.271523 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs"] Oct 30 22:26:13.271774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.271749 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" podUID="02787274-f04a-491f-bc8f-e340311f3ac7" containerName="maas-api" containerID="cri-o://cb6485f6f5708da67f75288c3f6b8e88cfdfe52df118888d5f3ff7455d71ead9" gracePeriod=30 Oct 30 22:26:13.346026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.341795 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcrm\" (UniqueName: \"kubernetes.io/projected/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a-kube-api-access-mjcrm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.367335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.367308 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:26:13.442331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.442289 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume\") pod \"6c8b05ae-b30f-472d-9658-912a7ec57a64\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " Oct 30 22:26:13.442529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.442340 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv\" (UniqueName: \"kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv\") pod \"6c8b05ae-b30f-472d-9658-912a7ec57a64\" (UID: \"6c8b05ae-b30f-472d-9658-912a7ec57a64\") " Oct 30 22:26:13.442753 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.442730 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume" pod "6c8b05ae-b30f-472d-9658-912a7ec57a64" (UID: "6c8b05ae-b30f-472d-9658-912a7ec57a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:13.442933 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.442919 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8b05ae-b30f-472d-9658-912a7ec57a64-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.444701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.444678 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv" pod "6c8b05ae-b30f-472d-9658-912a7ec57a64" (UID: "6c8b05ae-b30f-472d-9658-912a7ec57a64"). InnerVolumeSpecName "kube-api-access-rpbvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:13.520028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.519988 5607 generic.go:356] "Generic (PLEG): container finished" podID="a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" containerID="483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878" exitCode=0 Oct 30 22:26:13.520301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.520106 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" event={"ID":"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a","Type":"ContainerDied","Data":"483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878"} Oct 30 22:26:13.520301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.520160 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" event={"ID":"a42cfd2b-0420-4e98-b778-6f0c80ec1a4a","Type":"ContainerDied","Data":"9649049f9dcf33307106105ac4c75c1e43a1d06374d28165132e2ddf2d3d3ec3"} Oct 30 22:26:13.520301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.520188 5607 scope.go:122] "RemoveContainer" containerID="483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878" Oct 30 22:26:13.520543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.520321 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl" Oct 30 22:26:13.540388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.540353 5607 generic.go:356] "Generic (PLEG): container finished" podID="6c8b05ae-b30f-472d-9658-912a7ec57a64" containerID="f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777" exitCode=0 Oct 30 22:26:13.540570 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.540555 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" Oct 30 22:26:13.541247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.541212 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" event={"ID":"6c8b05ae-b30f-472d-9658-912a7ec57a64","Type":"ContainerDied","Data":"f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777"} Oct 30 22:26:13.541358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.541273 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t" event={"ID":"6c8b05ae-b30f-472d-9658-912a7ec57a64","Type":"ContainerDied","Data":"dd3e7b593328f0dbc39911c6d981e5d629b4c29dcb94f0f30c2cbd9b4f3fd006"} Oct 30 22:26:13.548645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.548592 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbvv\" (UniqueName: \"kubernetes.io/projected/6c8b05ae-b30f-472d-9658-912a7ec57a64-kube-api-access-rpbvv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.555595 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.555560 5607 generic.go:356] "Generic (PLEG): container finished" podID="02787274-f04a-491f-bc8f-e340311f3ac7" containerID="cb6485f6f5708da67f75288c3f6b8e88cfdfe52df118888d5f3ff7455d71ead9" exitCode=0 Oct 30 22:26:13.555738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.555619 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" event={"ID":"02787274-f04a-491f-bc8f-e340311f3ac7","Type":"ContainerDied","Data":"cb6485f6f5708da67f75288c3f6b8e88cfdfe52df118888d5f3ff7455d71ead9"} Oct 30 22:26:13.557842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.557825 5607 scope.go:122] "RemoveContainer" containerID="483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878" Oct 30 22:26:13.558641 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:13.558479 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878\": container with ID starting with 483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878 not found: ID does not exist" containerID="483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878" Oct 30 22:26:13.558641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.558516 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878"} err="failed to get container status \"483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878\": rpc error: code = NotFound desc = could not find container \"483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878\": container with ID starting with 483ae51744f16a7bc4979a61033ed7035507dbee6635a3a782ea79212d328878 not found: ID does not exist" Oct 30 22:26:13.558641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.558537 5607 scope.go:122] "RemoveContainer" containerID="f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777" Oct 30 22:26:13.571086 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.571060 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:26:13.572362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.572340 5607 scope.go:122] "RemoveContainer" containerID="f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777" Oct 30 22:26:13.572647 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:13.572627 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777\": container with ID starting with f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777 not found: ID does not exist" containerID="f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777" Oct 30 22:26:13.572698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.572656 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777"} err="failed to get container status \"f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777\": rpc error: code = NotFound desc = could not find container \"f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777\": container with ID starting with f9a3c3a6d455ab1925af7c3db8f4d30c0ab0f270580da6d5a8aaa9f849e10777 not found: ID does not exist" Oct 30 22:26:13.651436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.651406 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls\") pod \"02787274-f04a-491f-bc8f-e340311f3ac7\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " Oct 30 22:26:13.651630 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.651606 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd\" (UniqueName: \"kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd\") pod \"02787274-f04a-491f-bc8f-e340311f3ac7\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " Oct 30 22:26:13.652813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.652783 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca\") pod \"02787274-f04a-491f-bc8f-e340311f3ac7\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " Oct 30 22:26:13.652985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.652928 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs\") pod \"02787274-f04a-491f-bc8f-e340311f3ac7\" (UID: \"02787274-f04a-491f-bc8f-e340311f3ac7\") " Oct 30 22:26:13.653502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.653378 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca" pod "02787274-f04a-491f-bc8f-e340311f3ac7" (UID: "02787274-f04a-491f-bc8f-e340311f3ac7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:13.653666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.653560 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02787274-f04a-491f-bc8f-e340311f3ac7-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.654561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.654537 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls" pod "02787274-f04a-491f-bc8f-e340311f3ac7" (UID: "02787274-f04a-491f-bc8f-e340311f3ac7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:13.655766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.655721 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl"] Oct 30 22:26:13.656055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.655930 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs" pod "02787274-f04a-491f-bc8f-e340311f3ac7" (UID: "02787274-f04a-491f-bc8f-e340311f3ac7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:13.657098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.657076 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd" pod "02787274-f04a-491f-bc8f-e340311f3ac7" (UID: "02787274-f04a-491f-bc8f-e340311f3ac7"). InnerVolumeSpecName "kube-api-access-l6ddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:13.659282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.659258 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c89e47-865cb6cd4c-46vtl"] Oct 30 22:26:13.755162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.755130 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.755162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.755159 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6ddd\" (UniqueName: \"kubernetes.io/projected/02787274-f04a-491f-bc8f-e340311f3ac7-kube-api-access-l6ddd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.755162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.755170 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02787274-f04a-491f-bc8f-e340311f3ac7-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:13.891696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.891611 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t"] Oct 30 22:26:13.895637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.895613 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c89e47-68c7b5fd55-zkb2t"] Oct 30 22:26:13.949972 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:13.949930 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4"] Oct 30 22:26:14.056342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.056312 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv"] Oct 30 22:26:14.563206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.563113 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" Oct 30 22:26:14.563206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.563112 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs" event={"ID":"02787274-f04a-491f-bc8f-e340311f3ac7","Type":"ContainerDied","Data":"9498bc58cc9f5f53d92342c4ec82e5d3c65c4c35dd1a49a14ff2cad627f7e111"} Oct 30 22:26:14.563761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.563276 5607 scope.go:122] "RemoveContainer" containerID="cb6485f6f5708da67f75288c3f6b8e88cfdfe52df118888d5f3ff7455d71ead9" Oct 30 22:26:14.732179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.732142 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs"] Oct 30 22:26:14.736395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.736364 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0e90fe-bf67c9549-rm7hs"] Oct 30 22:26:14.871763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.871672 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02787274-f04a-491f-bc8f-e340311f3ac7" path="/var/lib/kubelet/pods/02787274-f04a-491f-bc8f-e340311f3ac7/volumes" Oct 30 22:26:14.872263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.872241 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2740c3d2-b33f-4e20-9deb-6538a4e9413f" path="/var/lib/kubelet/pods/2740c3d2-b33f-4e20-9deb-6538a4e9413f/volumes" Oct 30 22:26:14.872714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.872695 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8b05ae-b30f-472d-9658-912a7ec57a64" path="/var/lib/kubelet/pods/6c8b05ae-b30f-472d-9658-912a7ec57a64/volumes" Oct 30 22:26:14.873448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:14.873428 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" path="/var/lib/kubelet/pods/a42cfd2b-0420-4e98-b778-6f0c80ec1a4a/volumes" Oct 30 22:26:18.951122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:18.951081 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" podUID="7e74e7de-63e0-469f-bf3d-0978f98e3182" containerName="payload-processing" containerID="cri-o://3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c" gracePeriod=25 Oct 30 22:26:19.057380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.057344 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" podUID="0f1b72d4-0add-4ea4-acae-6a90fbbafce5" containerName="payload-pre-processing" containerID="cri-o://cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1" gracePeriod=25 Oct 30 22:26:19.233826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.233801 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:26:19.244467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.244155 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8\" (UniqueName: \"kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8\") pod \"7e74e7de-63e0-469f-bf3d-0978f98e3182\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " Oct 30 22:26:19.244467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.244236 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume\") pod \"7e74e7de-63e0-469f-bf3d-0978f98e3182\" (UID: \"7e74e7de-63e0-469f-bf3d-0978f98e3182\") " Oct 30 22:26:19.245063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.245031 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume" pod "7e74e7de-63e0-469f-bf3d-0978f98e3182" (UID: "7e74e7de-63e0-469f-bf3d-0978f98e3182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:19.247533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.247508 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8" pod "7e74e7de-63e0-469f-bf3d-0978f98e3182" (UID: "7e74e7de-63e0-469f-bf3d-0978f98e3182"). InnerVolumeSpecName "kube-api-access-dkcp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:19.327753 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.327731 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:26:19.345909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.345884 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf\" (UniqueName: \"kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf\") pod \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " Oct 30 22:26:19.346117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.346102 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume\") pod \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\" (UID: \"0f1b72d4-0add-4ea4-acae-6a90fbbafce5\") " Oct 30 22:26:19.346434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.346411 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcp8\" (UniqueName: \"kubernetes.io/projected/7e74e7de-63e0-469f-bf3d-0978f98e3182-kube-api-access-dkcp8\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:19.346434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.346433 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74e7de-63e0-469f-bf3d-0978f98e3182-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:19.346575 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.346430 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume" pod "0f1b72d4-0add-4ea4-acae-6a90fbbafce5" (UID: "0f1b72d4-0add-4ea4-acae-6a90fbbafce5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:19.347975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.347938 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf" pod "0f1b72d4-0add-4ea4-acae-6a90fbbafce5" (UID: "0f1b72d4-0add-4ea4-acae-6a90fbbafce5"). InnerVolumeSpecName "kube-api-access-nf5lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:19.447853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.447817 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:19.447853 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.447850 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5lf\" (UniqueName: \"kubernetes.io/projected/0f1b72d4-0add-4ea4-acae-6a90fbbafce5-kube-api-access-nf5lf\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:19.656619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.656521 5607 generic.go:356] "Generic (PLEG): container finished" podID="0f1b72d4-0add-4ea4-acae-6a90fbbafce5" containerID="cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1" exitCode=0 Oct 30 22:26:19.656619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.656560 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" event={"ID":"0f1b72d4-0add-4ea4-acae-6a90fbbafce5","Type":"ContainerDied","Data":"cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1"} Oct 30 22:26:19.656619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.656588 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" Oct 30 22:26:19.656918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.656647 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv" event={"ID":"0f1b72d4-0add-4ea4-acae-6a90fbbafce5","Type":"ContainerDied","Data":"19323e829062bf3e7a0809d96f5349f5f5baf7b238af8d3b79b48a9d7fb5b01a"} Oct 30 22:26:19.656918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.656679 5607 scope.go:122] "RemoveContainer" containerID="cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1" Oct 30 22:26:19.658350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.658323 5607 generic.go:356] "Generic (PLEG): container finished" podID="7e74e7de-63e0-469f-bf3d-0978f98e3182" containerID="3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c" exitCode=0 Oct 30 22:26:19.658510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.658390 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" event={"ID":"7e74e7de-63e0-469f-bf3d-0978f98e3182","Type":"ContainerDied","Data":"3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c"} Oct 30 22:26:19.658510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.658431 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" event={"ID":"7e74e7de-63e0-469f-bf3d-0978f98e3182","Type":"ContainerDied","Data":"1f7c72b258b7bb8f08677a2f1e01e79244124b75cf7a9173d891896f603a9b80"} Oct 30 22:26:19.658510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.658467 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4" Oct 30 22:26:19.668839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.668815 5607 scope.go:122] "RemoveContainer" containerID="cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1" Oct 30 22:26:19.669175 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:19.669150 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1\": container with ID starting with cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1 not found: ID does not exist" containerID="cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1" Oct 30 22:26:19.669341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.669187 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1"} err="failed to get container status \"cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1\": rpc error: code = NotFound desc = could not find container \"cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1\": container with ID starting with cf3b483d844cd534e6a64d0e96aaa41d807a175ebcf226c024e4f16459c3acb1 not found: ID does not exist" Oct 30 22:26:19.669341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.669208 5607 scope.go:122] "RemoveContainer" containerID="3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c" Oct 30 22:26:19.678593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.678573 5607 scope.go:122] "RemoveContainer" containerID="3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c" Oct 30 22:26:19.678869 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:19.678845 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c\": container with ID starting with 3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c not found: ID does not exist" containerID="3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c" Oct 30 22:26:19.678988 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.678875 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c"} err="failed to get container status \"3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c\": rpc error: code = NotFound desc = could not find container \"3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c\": container with ID starting with 3cf4deb064118310c54ad1c3ab1fb367ee25fa4625f77f4a62b6829fc4ab2e4c not found: ID does not exist" Oct 30 22:26:19.828053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.828020 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv"] Oct 30 22:26:19.832569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.832539 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ebcc6d-5999fdd579-gpdwv"] Oct 30 22:26:19.945171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.945140 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4"] Oct 30 22:26:19.949788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:19.949761 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ebcc6d-787f588799-sb9m4"] Oct 30 22:26:20.871476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:20.871444 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1b72d4-0add-4ea4-acae-6a90fbbafce5" path="/var/lib/kubelet/pods/0f1b72d4-0add-4ea4-acae-6a90fbbafce5/volumes" Oct 30 22:26:20.871905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:20.871891 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e74e7de-63e0-469f-bf3d-0978f98e3182" path="/var/lib/kubelet/pods/7e74e7de-63e0-469f-bf3d-0978f98e3182/volumes" Oct 30 22:26:21.724898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:21.723565 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8"] Oct 30 22:26:21.724898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:21.723829 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" containerID="cri-o://42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8" gracePeriod=30 Oct 30 22:26:22.864580 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:22.864540 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:26:23.274531 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:23.274490 5607 patch_prober.go:29] interesting pod/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:23.274714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:23.274591 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:27.121004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.120946 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:26:27.302227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.302188 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.302393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.302242 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.302613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.302583 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.302714 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.302695 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.302794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.302777 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.302978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.302873 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.303320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.303195 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:27.303513 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.303494 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.303616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.303588 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.303681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.303633 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.303681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.303664 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg\") pod \"144cd0b3-1530-4807-b04f-230c95bbce81\" (UID: \"144cd0b3-1530-4807-b04f-230c95bbce81\") " Oct 30 22:26:27.303915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.303883 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.304226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.304129 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.304422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.304396 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.304535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.304426 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.304535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.304443 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.304535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.304456 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144cd0b3-1530-4807-b04f-230c95bbce81-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.304730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.304489 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.305082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.305054 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:27.305697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.305673 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:27.306222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.306193 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:27.307261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.307241 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg" pod "144cd0b3-1530-4807-b04f-230c95bbce81" (UID: "144cd0b3-1530-4807-b04f-230c95bbce81"). InnerVolumeSpecName "kube-api-access-zlhpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:27.406179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.406094 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.406179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.406123 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144cd0b3-1530-4807-b04f-230c95bbce81-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.406179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.406137 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.406179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.406149 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhpg\" (UniqueName: \"kubernetes.io/projected/144cd0b3-1530-4807-b04f-230c95bbce81-kube-api-access-zlhpg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.406179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.406161 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144cd0b3-1530-4807-b04f-230c95bbce81-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:27.740661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.740627 5607 generic.go:356] "Generic (PLEG): container finished" podID="144cd0b3-1530-4807-b04f-230c95bbce81" containerID="42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8" exitCode=0 Oct 30 22:26:27.740864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.740676 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" event={"ID":"144cd0b3-1530-4807-b04f-230c95bbce81","Type":"ContainerDied","Data":"42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8"} Oct 30 22:26:27.740864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.740678 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" Oct 30 22:26:27.740864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.740718 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8" event={"ID":"144cd0b3-1530-4807-b04f-230c95bbce81","Type":"ContainerDied","Data":"970bb4417c529cb3997bceb954a89ffd9654f0744906db21fe8403aa9a3ae02d"} Oct 30 22:26:27.740864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.740738 5607 scope.go:122] "RemoveContainer" containerID="42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8" Oct 30 22:26:27.754168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.754134 5607 scope.go:122] "RemoveContainer" containerID="c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b" Oct 30 22:26:27.764046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.764025 5607 scope.go:122] "RemoveContainer" containerID="42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8" Oct 30 22:26:27.764340 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:27.764319 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8\": container with ID starting with 42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8 not found: ID does not exist" containerID="42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8" Oct 30 22:26:27.764419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.764352 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8"} err="failed to get container status \"42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8\": rpc error: code = NotFound desc = could not find container \"42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8\": container with ID starting with 42c66608edc9ba3b261c519753e444ae85969348275a6055d5ee696f9a6cacf8 not found: ID does not exist" Oct 30 22:26:27.764419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.764377 5607 scope.go:122] "RemoveContainer" containerID="c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b" Oct 30 22:26:27.764639 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:27.764618 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b\": container with ID starting with c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b not found: ID does not exist" containerID="c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b" Oct 30 22:26:27.764701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.764648 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b"} err="failed to get container status \"c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b\": rpc error: code = NotFound desc = could not find container \"c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b\": container with ID starting with c50e1cb32f572e7dac303055a7343c8201a13f3409fc79eb6b5305cce71d020b not found: ID does not exist" Oct 30 22:26:27.928071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.928033 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8"] Oct 30 22:26:27.980706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:27.980672 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-40fb42-openshift-default-6858979c77-hbgs8"] Oct 30 22:26:28.871723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:28.871687 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" path="/var/lib/kubelet/pods/144cd0b3-1530-4807-b04f-230c95bbce81/volumes" Oct 30 22:26:34.191361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:34.191326 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk"] Oct 30 22:26:35.537893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.537863 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg"] Oct 30 22:26:35.538315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.538087 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" podUID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" containerName="maas-api" containerID="cri-o://9fb69e1ce9a7f6323c4e61b2d81d8c4b92c0d7afaf95681ee983815e64089305" gracePeriod=30 Oct 30 22:26:35.836135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.834793 5607 generic.go:356] "Generic (PLEG): container finished" podID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" containerID="9fb69e1ce9a7f6323c4e61b2d81d8c4b92c0d7afaf95681ee983815e64089305" exitCode=0 Oct 30 22:26:35.836135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.834910 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" event={"ID":"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5","Type":"ContainerDied","Data":"9fb69e1ce9a7f6323c4e61b2d81d8c4b92c0d7afaf95681ee983815e64089305"} Oct 30 22:26:35.888774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.888750 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:26:35.911237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.911213 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9\" (UniqueName: \"kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9\") pod \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " Oct 30 22:26:35.911388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.911375 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\") pod \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " Oct 30 22:26:35.911432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.911417 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca\") pod \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " Oct 30 22:26:35.911484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.911473 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs\") pod \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\" (UID: \"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5\") " Oct 30 22:26:35.911849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.911818 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca" pod "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" (UID: "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:35.912014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.912000 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:35.913611 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.913580 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9" pod "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" (UID: "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5"). InnerVolumeSpecName "kube-api-access-nk8q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:35.914138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.914119 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs" pod "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" (UID: "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:35.914223 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:35.914138 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls" pod "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" (UID: "eddab44b-4e7e-4c7c-ae75-fddff06f2fd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:36.013598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:36.013572 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.013598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:36.013596 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.013746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:36.013606 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk8q9\" (UniqueName: \"kubernetes.io/projected/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5-kube-api-access-nk8q9\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:36.843248 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:36.843221 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" Oct 30 22:26:36.843674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:36.843255 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" event={"ID":"eddab44b-4e7e-4c7c-ae75-fddff06f2fd5","Type":"ContainerDied","Data":"32c8ec5fabd6174c3e69575800abddc767356ea89530991cd34213c7c20fac3b"} Oct 30 22:26:36.843674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:36.843313 5607 scope.go:122] "RemoveContainer" containerID="9fb69e1ce9a7f6323c4e61b2d81d8c4b92c0d7afaf95681ee983815e64089305" Oct 30 22:26:37.013028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:37.012999 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg"] Oct 30 22:26:37.063449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:37.063415 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg"] Oct 30 22:26:38.716535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:38.716495 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-872f4a-5476b97b96-hlrbg" podUID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.177:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:38.870872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:38.870842 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" path="/var/lib/kubelet/pods/eddab44b-4e7e-4c7c-ae75-fddff06f2fd5/volumes" Oct 30 22:26:39.192557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.192515 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" podUID="da1d28d0-d062-4382-87ea-cdf2059251d0" containerName="payload-processing" containerID="cri-o://59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d" gracePeriod=25 Oct 30 22:26:39.471931 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.471909 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:26:39.553238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.553207 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr\" (UniqueName: \"kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr\") pod \"da1d28d0-d062-4382-87ea-cdf2059251d0\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " Oct 30 22:26:39.553354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.553253 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume\") pod \"da1d28d0-d062-4382-87ea-cdf2059251d0\" (UID: \"da1d28d0-d062-4382-87ea-cdf2059251d0\") " Oct 30 22:26:39.553668 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.553642 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume" pod "da1d28d0-d062-4382-87ea-cdf2059251d0" (UID: "da1d28d0-d062-4382-87ea-cdf2059251d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:39.555589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.555569 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr" pod "da1d28d0-d062-4382-87ea-cdf2059251d0" (UID: "da1d28d0-d062-4382-87ea-cdf2059251d0"). InnerVolumeSpecName "kube-api-access-zdtzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:39.654478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.654453 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdtzr\" (UniqueName: \"kubernetes.io/projected/da1d28d0-d062-4382-87ea-cdf2059251d0-kube-api-access-zdtzr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:39.654478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.654481 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1d28d0-d062-4382-87ea-cdf2059251d0-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:39.868427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.868342 5607 generic.go:356] "Generic (PLEG): container finished" podID="da1d28d0-d062-4382-87ea-cdf2059251d0" containerID="59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d" exitCode=0 Oct 30 22:26:39.868427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.868404 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" Oct 30 22:26:39.869023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.868434 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" event={"ID":"da1d28d0-d062-4382-87ea-cdf2059251d0","Type":"ContainerDied","Data":"59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d"} Oct 30 22:26:39.869023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.868489 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk" event={"ID":"da1d28d0-d062-4382-87ea-cdf2059251d0","Type":"ContainerDied","Data":"ca13a70d98ff5fb8f86df6adf3ff8308b8dba2f6eab451de98320f3b80d50153"} Oct 30 22:26:39.869023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.868510 5607 scope.go:122] "RemoveContainer" containerID="59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d" Oct 30 22:26:39.879110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.879042 5607 scope.go:122] "RemoveContainer" containerID="59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d" Oct 30 22:26:39.879336 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:39.879317 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d\": container with ID starting with 59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d not found: ID does not exist" containerID="59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d" Oct 30 22:26:39.879389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:39.879347 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d"} err="failed to get container status \"59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d\": rpc error: code = NotFound desc = could not find container \"59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d\": container with ID starting with 59b39d84b3cb9dac42f53e170fc7c8f3a9dbc8de6dd5f644c2ae0167fc7a8a1d not found: ID does not exist" Oct 30 22:26:40.037436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:40.037404 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk"] Oct 30 22:26:40.088155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:40.088128 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-923d78-5cdf899975-56sdk"] Oct 30 22:26:40.870490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:40.870457 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1d28d0-d062-4382-87ea-cdf2059251d0" path="/var/lib/kubelet/pods/da1d28d0-d062-4382-87ea-cdf2059251d0/volumes" Oct 30 22:26:52.864482 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:26:52.864442 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:26:54.275038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:54.275005 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq"] Oct 30 22:26:54.275522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:54.275234 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" containerID="cri-o://9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb" gracePeriod=30 Oct 30 22:26:59.040971 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.040928 5607 patch_prober.go:29] interesting pod/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:59.041421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.041013 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:59.546680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.546629 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:26:59.680376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680345 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680392 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680430 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680466 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680493 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680541 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680616 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680705 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680781 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.680935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680797 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.680935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680821 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket\") pod \"2aa5eb97-61b8-49de-8499-491918f935a7\" (UID: \"2aa5eb97-61b8-49de-8499-491918f935a7\") " Oct 30 22:26:59.680935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.680825 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:59.681191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681003 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.681264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681186 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.681558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681535 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.681716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681700 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.681843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681812 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa5eb97-61b8-49de-8499-491918f935a7-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.681843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681837 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.682049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.681849 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.683253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.683232 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "kube-api-access-844ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:59.683436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.683417 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:59.683490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.683454 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:59.684321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.684304 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo" pod "2aa5eb97-61b8-49de-8499-491918f935a7" (UID: "2aa5eb97-61b8-49de-8499-491918f935a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:59.783301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.783263 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa5eb97-61b8-49de-8499-491918f935a7-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.783301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.783292 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.783484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.783306 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844ls\" (UniqueName: \"kubernetes.io/projected/2aa5eb97-61b8-49de-8499-491918f935a7-kube-api-access-844ls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:59.783484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:26:59.783321 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa5eb97-61b8-49de-8499-491918f935a7-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:00.002503 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.002469 5607 generic.go:356] "Generic (PLEG): container finished" podID="2aa5eb97-61b8-49de-8499-491918f935a7" containerID="9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb" exitCode=0 Oct 30 22:27:00.002705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.002554 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" Oct 30 22:27:00.002705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.002552 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" event={"ID":"2aa5eb97-61b8-49de-8499-491918f935a7","Type":"ContainerDied","Data":"9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb"} Oct 30 22:27:00.002705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.002696 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq" event={"ID":"2aa5eb97-61b8-49de-8499-491918f935a7","Type":"ContainerDied","Data":"8f2dbc0a1dc6d94310dd936ca0d3f73e6f1dc0a479630b21463ebf023ce7cc36"} Oct 30 22:27:00.002882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.002718 5607 scope.go:122] "RemoveContainer" containerID="9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb" Oct 30 22:27:00.013851 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.013833 5607 scope.go:122] "RemoveContainer" containerID="6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656" Oct 30 22:27:00.023006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.022987 5607 scope.go:122] "RemoveContainer" containerID="9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb" Oct 30 22:27:00.023296 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:00.023276 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb\": container with ID starting with 9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb not found: ID does not exist" containerID="9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb" Oct 30 22:27:00.023394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.023303 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb"} err="failed to get container status \"9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb\": rpc error: code = NotFound desc = could not find container \"9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb\": container with ID starting with 9556ad18be145b8300a4a4be6d91512c398b49f767b05d67180a2de5643990bb not found: ID does not exist" Oct 30 22:27:00.023394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.023318 5607 scope.go:122] "RemoveContainer" containerID="6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656" Oct 30 22:27:00.023566 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:00.023551 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656\": container with ID starting with 6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656 not found: ID does not exist" containerID="6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656" Oct 30 22:27:00.023615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.023570 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656"} err="failed to get container status \"6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656\": rpc error: code = NotFound desc = could not find container \"6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656\": container with ID starting with 6e8d0ee9e663a43bd3949423248c750ecf56fc71eb1cdde5086f3813d656b656 not found: ID does not exist" Oct 30 22:27:00.171550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.171526 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq"] Oct 30 22:27:00.222906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.222877 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0e90fe-openshift-default-5d495cbd97-vg6bq"] Oct 30 22:27:00.870413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:00.870384 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" path="/var/lib/kubelet/pods/2aa5eb97-61b8-49de-8499-491918f935a7/volumes" Oct 30 22:27:01.212355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:01.212329 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:27:01.214136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:01.214115 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:27:06.264586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:06.264553 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd"] Oct 30 22:27:06.265017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:06.264803 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" containerID="cri-o://4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9" gracePeriod=30 Oct 30 22:27:11.559545 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.559520 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740370 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740418 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740465 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740514 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740544 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740573 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740652 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740685 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.741696 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.740766 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo\") pod \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\" (UID: \"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf\") " Oct 30 22:27:11.747627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.747434 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:11.747840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.747804 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:11.748096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.748077 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:11.748289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.748269 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:11.754203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.752471 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:11.754405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.754384 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:11.759315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.759282 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:11.759315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.759303 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "kube-api-access-lsfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:11.760621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.760589 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy" pod "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" (UID: "afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:11.842669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842609 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842635 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842646 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842655 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842663 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842671 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842681 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842690 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfdv\" (UniqueName: \"kubernetes.io/projected/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-kube-api-access-lsfdv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.842916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:11.842700 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.074723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.074686 5607 generic.go:356] "Generic (PLEG): container finished" podID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerID="4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9" exitCode=0 Oct 30 22:27:12.074860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.074743 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" Oct 30 22:27:12.074860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.074765 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" event={"ID":"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf","Type":"ContainerDied","Data":"4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9"} Oct 30 22:27:12.074860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.074817 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd" event={"ID":"afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf","Type":"ContainerDied","Data":"5ed70ea812b7e96a102963912d67f1fd4ff4c3a0adcee688942d3bfae832aa0d"} Oct 30 22:27:12.074860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.074853 5607 scope.go:122] "RemoveContainer" containerID="4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9" Oct 30 22:27:12.085788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.085765 5607 scope.go:122] "RemoveContainer" containerID="d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411" Oct 30 22:27:12.095399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.095384 5607 scope.go:122] "RemoveContainer" containerID="4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9" Oct 30 22:27:12.095656 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:12.095636 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9\": container with ID starting with 4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9 not found: ID does not exist" containerID="4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9" Oct 30 22:27:12.095704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.095665 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9"} err="failed to get container status \"4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9\": rpc error: code = NotFound desc = could not find container \"4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9\": container with ID starting with 4a9fcc6aa43dbeb1d98da24261fd846298c393a74b6af1bd6e9e10c3d20124f9 not found: ID does not exist" Oct 30 22:27:12.095704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.095682 5607 scope.go:122] "RemoveContainer" containerID="d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411" Oct 30 22:27:12.095928 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:12.095911 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411\": container with ID starting with d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411 not found: ID does not exist" containerID="d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411" Oct 30 22:27:12.096024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.095932 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411"} err="failed to get container status \"d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411\": rpc error: code = NotFound desc = could not find container \"d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411\": container with ID starting with d57b0c10930a6fa5aec22ab6773ca67b64febf8cd2a372b04fb165b5ab15c411 not found: ID does not exist" Oct 30 22:27:12.246192 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.246168 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd"] Oct 30 22:27:12.250422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.250404 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c89e47-openshift-default-75d6698785-ggftd"] Oct 30 22:27:12.870364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:12.870331 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" path="/var/lib/kubelet/pods/afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf/volumes" Oct 30 22:27:13.712796 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:13.712753 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm"] Oct 30 22:27:13.713094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:13.713064 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" podUID="f355f307-0c39-4050-8cec-ba3f56a0234a" containerName="maas-api" containerID="cri-o://318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f" gracePeriod=30 Oct 30 22:27:13.729306 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:13.729274 5607 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf355f307_0c39_4050_8cec_ba3f56a0234a.slice/crio-318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:27:14.011577 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.011543 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:27:14.065599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.065573 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx\" (UniqueName: \"kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx\") pod \"f355f307-0c39-4050-8cec-ba3f56a0234a\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " Oct 30 22:27:14.065740 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.065606 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca\") pod \"f355f307-0c39-4050-8cec-ba3f56a0234a\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " Oct 30 22:27:14.065740 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.065695 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs\") pod \"f355f307-0c39-4050-8cec-ba3f56a0234a\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " Oct 30 22:27:14.065869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.065749 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\") pod \"f355f307-0c39-4050-8cec-ba3f56a0234a\" (UID: \"f355f307-0c39-4050-8cec-ba3f56a0234a\") " Oct 30 22:27:14.066106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.066072 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca" pod "f355f307-0c39-4050-8cec-ba3f56a0234a" (UID: "f355f307-0c39-4050-8cec-ba3f56a0234a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:14.066296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.066276 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f355f307-0c39-4050-8cec-ba3f56a0234a-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:14.068057 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.068031 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs" pod "f355f307-0c39-4050-8cec-ba3f56a0234a" (UID: "f355f307-0c39-4050-8cec-ba3f56a0234a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:14.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.068112 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx" pod "f355f307-0c39-4050-8cec-ba3f56a0234a" (UID: "f355f307-0c39-4050-8cec-ba3f56a0234a"). InnerVolumeSpecName "kube-api-access-882hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:14.068171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.068138 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls" pod "f355f307-0c39-4050-8cec-ba3f56a0234a" (UID: "f355f307-0c39-4050-8cec-ba3f56a0234a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:14.089519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.089497 5607 generic.go:356] "Generic (PLEG): container finished" podID="f355f307-0c39-4050-8cec-ba3f56a0234a" containerID="318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f" exitCode=0 Oct 30 22:27:14.089640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.089542 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" Oct 30 22:27:14.089640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.089586 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" event={"ID":"f355f307-0c39-4050-8cec-ba3f56a0234a","Type":"ContainerDied","Data":"318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f"} Oct 30 22:27:14.089763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.089650 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm" event={"ID":"f355f307-0c39-4050-8cec-ba3f56a0234a","Type":"ContainerDied","Data":"e4b0a257553f558f7826f0dfe22e1dfa8ccfb6a521de3c5e0a2f5a940d723ed6"} Oct 30 22:27:14.089763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.089684 5607 scope.go:122] "RemoveContainer" containerID="318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f" Oct 30 22:27:14.099796 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.099777 5607 scope.go:122] "RemoveContainer" containerID="318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f" Oct 30 22:27:14.100065 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:14.100046 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f\": container with ID starting with 318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f not found: ID does not exist" containerID="318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f" Oct 30 22:27:14.100119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.100072 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f"} err="failed to get container status \"318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f\": rpc error: code = NotFound desc = could not find container \"318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f\": container with ID starting with 318c5eb0971718a5b423be8384d6db6632e4fecb676e21cbbf3f286d6f089a5f not found: ID does not exist" Oct 30 22:27:14.167103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.167059 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-882hx\" (UniqueName: \"kubernetes.io/projected/f355f307-0c39-4050-8cec-ba3f56a0234a-kube-api-access-882hx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:14.167103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.167094 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:14.167103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.167111 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f355f307-0c39-4050-8cec-ba3f56a0234a-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:14.251636 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.251556 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm"] Oct 30 22:27:14.255087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.255065 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-923d78-58b5f6686d-9vgkm"] Oct 30 22:27:14.869645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:14.869613 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f355f307-0c39-4050-8cec-ba3f56a0234a" path="/var/lib/kubelet/pods/f355f307-0c39-4050-8cec-ba3f56a0234a/volumes" Oct 30 22:27:16.955258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.955228 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh"] Oct 30 22:27:16.956100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956045 5607 cpu_manager.go:414] "containerMap: removing container" podUID="0f1b72d4-0add-4ea4-acae-6a90fbbafce5" containerName="payload-pre-processing" Oct 30 22:27:16.956100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956069 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1b72d4-0add-4ea4-acae-6a90fbbafce5" containerName="payload-pre-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956123 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2740c3d2-b33f-4e20-9deb-6538a4e9413f" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956133 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2740c3d2-b33f-4e20-9deb-6538a4e9413f" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956147 5607 cpu_manager.go:414] "containerMap: removing container" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956154 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956184 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" containerName="payload-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956191 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" containerName="payload-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956199 5607 cpu_manager.go:414] "containerMap: removing container" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956206 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956214 5607 cpu_manager.go:414] "containerMap: removing container" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956220 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956229 5607 cpu_manager.go:414] "containerMap: removing container" podUID="02787274-f04a-491f-bc8f-e340311f3ac7" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956235 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02787274-f04a-491f-bc8f-e340311f3ac7" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956246 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f355f307-0c39-4050-8cec-ba3f56a0234a" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956253 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f355f307-0c39-4050-8cec-ba3f56a0234a" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956265 5607 cpu_manager.go:414] "containerMap: removing container" podUID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956272 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" containerName="maas-api" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956289 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7e74e7de-63e0-469f-bf3d-0978f98e3182" containerName="payload-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956296 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e74e7de-63e0-469f-bf3d-0978f98e3182" containerName="payload-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956312 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6c8b05ae-b30f-472d-9658-912a7ec57a64" containerName="payload-pre-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956319 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8b05ae-b30f-472d-9658-912a7ec57a64" containerName="payload-pre-processing" Oct 30 22:27:16.956323 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956331 5607 cpu_manager.go:414] "containerMap: removing container" podUID="da1d28d0-d062-4382-87ea-cdf2059251d0" containerName="payload-processing" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956338 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1d28d0-d062-4382-87ea-cdf2059251d0" containerName="payload-processing" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956365 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956372 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956384 5607 cpu_manager.go:414] "containerMap: removing container" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956391 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956406 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956414 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956551 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8b05ae-b30f-472d-9658-912a7ec57a64" containerName="payload-pre-processing" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956564 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1d28d0-d062-4382-87ea-cdf2059251d0" containerName="payload-processing" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956579 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956586 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956595 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1b72d4-0add-4ea4-acae-6a90fbbafce5" containerName="payload-pre-processing" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956606 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2740c3d2-b33f-4e20-9deb-6538a4e9413f" containerName="maas-api" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956618 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956630 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42cfd2b-0420-4e98-b778-6f0c80ec1a4a" containerName="payload-processing" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956640 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="144cd0b3-1530-4807-b04f-230c95bbce81" containerName="istio-proxy" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956649 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="02787274-f04a-491f-bc8f-e340311f3ac7" containerName="maas-api" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956657 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f355f307-0c39-4050-8cec-ba3f56a0234a" containerName="maas-api" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956665 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="eddab44b-4e7e-4c7c-ae75-fddff06f2fd5" containerName="maas-api" Oct 30 22:27:16.957413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.956675 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e74e7de-63e0-469f-bf3d-0978f98e3182" containerName="payload-processing" Oct 30 22:27:16.961883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:16.961862 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.001349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.001315 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5nv\" (UniqueName: \"kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv\") pod \"payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.001447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.001364 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.021243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.021216 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-459aa8\"" Oct 30 22:27:17.071781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.071760 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-459aa8-dockercfg-2ndvh\"" Oct 30 22:27:17.075942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.075917 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh"] Oct 30 22:27:17.102774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.102751 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5nv\" (UniqueName: \"kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv\") pod \"payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.102906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.102785 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.103508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.103487 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.115075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.115048 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4"] Oct 30 22:27:17.116143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.116120 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa5eb97-61b8-49de-8499-491918f935a7" containerName="istio-proxy" Oct 30 22:27:17.116296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.116277 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcd4f8f-3f3b-46f3-b4fe-36bc4fc25fcf" containerName="istio-proxy" Oct 30 22:27:17.121091 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.121043 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.166031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.166003 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5nv\" (UniqueName: \"kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv\") pod \"payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.177342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.177306 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4"] Oct 30 22:27:17.203701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.203671 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rn6w\" (UniqueName: \"kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w\") pod \"payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.203861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.203739 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.279296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.279228 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:17.305136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.305079 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rn6w\" (UniqueName: \"kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w\") pod \"payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.305282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.305139 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.305841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.305816 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.369749 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.367670 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rn6w\" (UniqueName: \"kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w\") pod \"payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.451095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.451064 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:17.649327 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:27:17.649294 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3c0fd0_9df5_4109_bc54_1951a0427430.slice/crio-f5c49525c7c82e2bdab281618d45d61e106c29442367922937b6c2b629695b19 WatchSource:0}: Error finding container f5c49525c7c82e2bdab281618d45d61e106c29442367922937b6c2b629695b19: Status 404 returned error can't find the container with id f5c49525c7c82e2bdab281618d45d61e106c29442367922937b6c2b629695b19 Oct 30 22:27:17.652444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.652377 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh"] Oct 30 22:27:17.823323 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:27:17.823294 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51ef35f_3e75_4950_8644_3e388e6ebac5.slice/crio-fe5e131f534e28efa3a17a692d1b7d35cbb992f9d7c6406b40d6d180081be15e WatchSource:0}: Error finding container fe5e131f534e28efa3a17a692d1b7d35cbb992f9d7c6406b40d6d180081be15e: Status 404 returned error can't find the container with id fe5e131f534e28efa3a17a692d1b7d35cbb992f9d7c6406b40d6d180081be15e Oct 30 22:27:17.826629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:17.826609 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4"] Oct 30 22:27:18.119226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.119195 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" event={"ID":"c51ef35f-3e75-4950-8644-3e388e6ebac5","Type":"ContainerStarted","Data":"d37de93cc2989b5e26606a5e1dc65dcb4b8e8b3dbeff9c812a326b170e0982aa"} Oct 30 22:27:18.119651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.119246 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" event={"ID":"c51ef35f-3e75-4950-8644-3e388e6ebac5","Type":"ContainerStarted","Data":"fe5e131f534e28efa3a17a692d1b7d35cbb992f9d7c6406b40d6d180081be15e"} Oct 30 22:27:18.119651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.119277 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:18.120757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.120737 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" event={"ID":"be3c0fd0-9df5-4109-bc54-1951a0427430","Type":"ContainerStarted","Data":"11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513"} Oct 30 22:27:18.120883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.120776 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" event={"ID":"be3c0fd0-9df5-4109-bc54-1951a0427430","Type":"ContainerStarted","Data":"f5c49525c7c82e2bdab281618d45d61e106c29442367922937b6c2b629695b19"} Oct 30 22:27:18.120883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.120814 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:18.238382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.238337 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" podStartSLOduration=1.238323121 podStartE2EDuration="1.238323121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:18.237840029 +0000 UTC m=+2417.638396149" watchObservedRunningTime="2026-10-30 22:27:18.238323121 +0000 UTC m=+2417.638879241" Oct 30 22:27:18.360909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:18.360849 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" podStartSLOduration=2.360832132 podStartE2EDuration="2.360832132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:18.355583594 +0000 UTC m=+2417.756139715" watchObservedRunningTime="2026-10-30 22:27:18.360832132 +0000 UTC m=+2417.761388253" Oct 30 22:27:22.863038 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:22.863003 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:27:23.350987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:23.350936 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6"] Oct 30 22:27:23.351249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:23.351207 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" containerID="cri-o://9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60" gracePeriod=30 Oct 30 22:27:26.580994 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:26.580931 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:27:26.581482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:26.581021 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:27:28.633598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.633575 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:27:28.728190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728138 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728289 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728376 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728417 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728447 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728543 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728585 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728640 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.728812 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.728675 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert\") pod \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\" (UID: \"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d\") " Oct 30 22:27:28.730258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.729766 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.730258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.729783 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.730258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.730119 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.730258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.730238 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.730591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.730385 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:28.735218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.733128 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:28.737056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.737014 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "kube-api-access-zsmg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:28.737307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.737269 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:28.737465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.737428 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo" pod "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" (UID: "6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:27:28.830663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830639 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830668 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830681 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830689 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830698 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830708 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830716 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830724 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsmg9\" (UniqueName: \"kubernetes.io/projected/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-kube-api-access-zsmg9\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:28.830797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:28.830734 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:29.130332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.130272 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:27:29.130476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.130339 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:27:29.205382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.205358 5607 generic.go:356] "Generic (PLEG): container finished" podID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerID="9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60" exitCode=0 Oct 30 22:27:29.205507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.205408 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" Oct 30 22:27:29.205507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.205434 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" event={"ID":"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d","Type":"ContainerDied","Data":"9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60"} Oct 30 22:27:29.205507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.205484 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6" event={"ID":"6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d","Type":"ContainerDied","Data":"2f95b206fdea2d4f408c7393d8eb4e58676059ab9f6e9e7b873c3b2e3b4ecf78"} Oct 30 22:27:29.205648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.205510 5607 scope.go:122] "RemoveContainer" containerID="9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60" Oct 30 22:27:29.215855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.215838 5607 scope.go:122] "RemoveContainer" containerID="10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21" Oct 30 22:27:29.225289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.225262 5607 scope.go:122] "RemoveContainer" containerID="9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60" Oct 30 22:27:29.225563 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:29.225544 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60\": container with ID starting with 9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60 not found: ID does not exist" containerID="9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60" Oct 30 22:27:29.225639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.225575 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60"} err="failed to get container status \"9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60\": rpc error: code = NotFound desc = could not find container \"9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60\": container with ID starting with 9d0c4e50e7318a0a5933b2305ff6f2f58f1d10e361102424e20f98da58ea9d60 not found: ID does not exist" Oct 30 22:27:29.225639 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.225598 5607 scope.go:122] "RemoveContainer" containerID="10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21" Oct 30 22:27:29.225852 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:29.225830 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21\": container with ID starting with 10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21 not found: ID does not exist" containerID="10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21" Oct 30 22:27:29.225902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.225860 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21"} err="failed to get container status \"10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21\": rpc error: code = NotFound desc = could not find container \"10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21\": container with ID starting with 10fa7c6774c9b5d69b33bfdc8a8ec1411f464fb9746c8a0bb4478298e4a78c21 not found: ID does not exist" Oct 30 22:27:29.486926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.486897 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6"] Oct 30 22:27:29.537470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:29.537438 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-923d78-openshift-default-b68d59d6f-wl9q6"] Oct 30 22:27:30.869885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:30.869851 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" path="/var/lib/kubelet/pods/6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d/volumes" Oct 30 22:27:36.441196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441163 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52"] Oct 30 22:27:36.441756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441741 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" Oct 30 22:27:36.441756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441756 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" Oct 30 22:27:36.441849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441766 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" Oct 30 22:27:36.441849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441771 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" Oct 30 22:27:36.441922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441868 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" Oct 30 22:27:36.441922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.441876 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a07bdb7-9b74-494d-855b-1a3f7d4b9d0d" containerName="istio-proxy" Oct 30 22:27:36.446884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.446867 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.504434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.504405 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52"] Oct 30 22:27:36.520004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.519979 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbb5h\" (UniqueName: \"kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h\") pod \"payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.520338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.520318 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.575354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.575312 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1875b4-dockercfg-9gpsb\"" Oct 30 22:27:36.575562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.575434 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1875b4\"" Oct 30 22:27:36.583267 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.583236 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b"] Oct 30 22:27:36.587364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.587346 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.622274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.622238 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbb5h\" (UniqueName: \"kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h\") pod \"payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.622432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.622372 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.623216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.623188 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.643573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.643526 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b"] Oct 30 22:27:36.702043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.701750 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbb5h\" (UniqueName: \"kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h\") pod \"payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.723102 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.723077 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgfrp\" (UniqueName: \"kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp\") pod \"payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.723251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.723218 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.761046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.761022 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:36.825692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.825656 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgfrp\" (UniqueName: \"kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp\") pod \"payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.825820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.825803 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.826761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.826726 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.890345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.890312 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgfrp\" (UniqueName: \"kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp\") pod \"payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:36.910540 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:36.910510 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:37.139053 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:27:37.139022 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d59d01_d737_4fc8_abba_6a3bca7d12bd.slice/crio-2f9dea9759a9777a17d4bfb3be34e33c3def59bf76615e5275fd0751ed2d8a0f WatchSource:0}: Error finding container 2f9dea9759a9777a17d4bfb3be34e33c3def59bf76615e5275fd0751ed2d8a0f: Status 404 returned error can't find the container with id 2f9dea9759a9777a17d4bfb3be34e33c3def59bf76615e5275fd0751ed2d8a0f Oct 30 22:27:37.142491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:37.142473 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52"] Oct 30 22:27:37.260651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:37.260560 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" event={"ID":"f7d59d01-d737-4fc8-abba-6a3bca7d12bd","Type":"ContainerStarted","Data":"c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e"} Oct 30 22:27:37.260651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:37.260624 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" event={"ID":"f7d59d01-d737-4fc8-abba-6a3bca7d12bd","Type":"ContainerStarted","Data":"2f9dea9759a9777a17d4bfb3be34e33c3def59bf76615e5275fd0751ed2d8a0f"} Oct 30 22:27:37.260839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:37.260654 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:37.289116 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:27:37.289086 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735721fb_43c5_4c59_833b_75666cdb2451.slice/crio-36cb4802121a4521384bec96421ebeb3997fd7fdbd744cddf7d4ba0ad6cfa518 WatchSource:0}: Error finding container 36cb4802121a4521384bec96421ebeb3997fd7fdbd744cddf7d4ba0ad6cfa518: Status 404 returned error can't find the container with id 36cb4802121a4521384bec96421ebeb3997fd7fdbd744cddf7d4ba0ad6cfa518 Oct 30 22:27:37.292467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:37.292447 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b"] Oct 30 22:27:37.379367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:37.379313 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" podStartSLOduration=1.379293929 podStartE2EDuration="1.379293929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:37.377771249 +0000 UTC m=+2436.778327371" watchObservedRunningTime="2026-10-30 22:27:37.379293929 +0000 UTC m=+2436.779850051" Oct 30 22:27:38.268013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:38.267972 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" event={"ID":"735721fb-43c5-4c59-833b-75666cdb2451","Type":"ContainerStarted","Data":"e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3"} Oct 30 22:27:38.268413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:38.268035 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" event={"ID":"735721fb-43c5-4c59-833b-75666cdb2451","Type":"ContainerStarted","Data":"36cb4802121a4521384bec96421ebeb3997fd7fdbd744cddf7d4ba0ad6cfa518"} Oct 30 22:27:38.268413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:38.268250 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:38.388243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:38.388192 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" podStartSLOduration=2.388179069 podStartE2EDuration="2.388179069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:38.386469335 +0000 UTC m=+2437.787025456" watchObservedRunningTime="2026-10-30 22:27:38.388179069 +0000 UTC m=+2437.788735187" Oct 30 22:27:47.520290 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.520206 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn"] Oct 30 22:27:47.524741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.524721 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.600819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.600790 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn"] Oct 30 22:27:47.639946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.639911 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4m7q\" (UniqueName: \"kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.640126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.640106 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.640318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.640290 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.647803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.647781 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:47.647922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.647806 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:27:47.741217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.741154 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.741422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.741299 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4m7q\" (UniqueName: \"kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.741422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.741395 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.742004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.741982 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.744077 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.744054 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.804477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.804406 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4m7q\" (UniqueName: \"kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q\") pod \"maas-api-revoke-sub-5ea3efd7-kkvbn\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:47.840082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:47.840044 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:48.233428 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:27:48.233389 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d552e1_110d_432e_8ad8_e56a669a8076.slice/crio-a6cceb3978f3623c1f27061d174a5302d0a626d4454c4e8e9c4dc83604d02156 WatchSource:0}: Error finding container a6cceb3978f3623c1f27061d174a5302d0a626d4454c4e8e9c4dc83604d02156: Status 404 returned error can't find the container with id a6cceb3978f3623c1f27061d174a5302d0a626d4454c4e8e9c4dc83604d02156 Oct 30 22:27:48.235333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:48.235307 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn"] Oct 30 22:27:48.271486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:48.271467 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:27:48.329973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:48.329934 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" event={"ID":"64d552e1-110d-432e-8ad8-e56a669a8076","Type":"ContainerStarted","Data":"a6cceb3978f3623c1f27061d174a5302d0a626d4454c4e8e9c4dc83604d02156"} Oct 30 22:27:49.276551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:49.276521 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:27:49.339203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:49.339138 5607 generic.go:356] "Generic (PLEG): container finished" podID="64d552e1-110d-432e-8ad8-e56a669a8076" containerID="a01e528c315f1ccb101ae9db044c07273776beed9a60952da3cc88174938f5c5" exitCode=0 Oct 30 22:27:49.339325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:49.339214 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" event={"ID":"64d552e1-110d-432e-8ad8-e56a669a8076","Type":"ContainerDied","Data":"a01e528c315f1ccb101ae9db044c07273776beed9a60952da3cc88174938f5c5"} Oct 30 22:27:50.496384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.496359 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:50.670467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.670394 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca\") pod \"64d552e1-110d-432e-8ad8-e56a669a8076\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " Oct 30 22:27:50.670627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.670467 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q\" (UniqueName: \"kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q\") pod \"64d552e1-110d-432e-8ad8-e56a669a8076\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " Oct 30 22:27:50.670627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.670569 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca\") pod \"64d552e1-110d-432e-8ad8-e56a669a8076\" (UID: \"64d552e1-110d-432e-8ad8-e56a669a8076\") " Oct 30 22:27:50.670915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.670892 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca" pod "64d552e1-110d-432e-8ad8-e56a669a8076" (UID: "64d552e1-110d-432e-8ad8-e56a669a8076"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:50.672686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.672665 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca" pod "64d552e1-110d-432e-8ad8-e56a669a8076" (UID: "64d552e1-110d-432e-8ad8-e56a669a8076"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:50.672769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.672736 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q" pod "64d552e1-110d-432e-8ad8-e56a669a8076" (UID: "64d552e1-110d-432e-8ad8-e56a669a8076"). InnerVolumeSpecName "kube-api-access-j4m7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:50.772360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.772332 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:50.772360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.772357 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d552e1-110d-432e-8ad8-e56a669a8076-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:50.772492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:50.772372 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4m7q\" (UniqueName: \"kubernetes.io/projected/64d552e1-110d-432e-8ad8-e56a669a8076-kube-api-access-j4m7q\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:27:51.362114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:51.362082 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" Oct 30 22:27:51.362114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:51.362108 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn" event={"ID":"64d552e1-110d-432e-8ad8-e56a669a8076","Type":"ContainerDied","Data":"a6cceb3978f3623c1f27061d174a5302d0a626d4454c4e8e9c4dc83604d02156"} Oct 30 22:27:51.362348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:51.362178 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6cceb3978f3623c1f27061d174a5302d0a626d4454c4e8e9c4dc83604d02156" Oct 30 22:27:52.863470 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:52.863432 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:27:55.476845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.476813 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s"] Oct 30 22:27:55.477430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.477415 5607 cpu_manager.go:414] "containerMap: removing container" podUID="64d552e1-110d-432e-8ad8-e56a669a8076" containerName="revoke-keys" Oct 30 22:27:55.477485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.477431 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d552e1-110d-432e-8ad8-e56a669a8076" containerName="revoke-keys" Oct 30 22:27:55.477561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.477552 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d552e1-110d-432e-8ad8-e56a669a8076" containerName="revoke-keys" Oct 30 22:27:55.480836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.480821 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.516882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.516854 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.517057 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.516918 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fx8k\" (UniqueName: \"kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.517057 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.517033 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.517235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.517214 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.587619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.587595 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-459aa8-serving-cert\"" Oct 30 22:27:55.590808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.590786 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s"] Oct 30 22:27:55.618380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.618344 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.618510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.618459 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.618584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.618513 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fx8k\" (UniqueName: \"kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.618649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.618629 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.618933 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:55.618754 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-serving-cert: secret "maas-api-e2e-praxis-a-459aa8-serving-cert" not found Oct 30 22:27:55.618933 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:27:55.618822 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls podName:0eb0bc20-8039-48d8-9ab3-6649188a7bf8 nodeName:}" failed. No retries permitted until 2026-10-30 22:27:56.118802379 +0000 UTC m=+2455.519358493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls") pod "maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" (UID: "0eb0bc20-8039-48d8-9ab3-6649188a7bf8") : secret "maas-api-e2e-praxis-a-459aa8-serving-cert" not found Oct 30 22:27:55.619357 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.619294 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.621252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.621219 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:55.681641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:55.681617 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fx8k\" (UniqueName: \"kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:56.129565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:56.129531 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:56.132361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:56.132329 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:56.393733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:56.393660 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:56.788010 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:27:56.787982 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb0bc20_8039_48d8_9ab3_6649188a7bf8.slice/crio-2a889b56b1b3b64454de18fa05c5c484f8909431aa3fabe8b57fe53c1ebbf2eb WatchSource:0}: Error finding container 2a889b56b1b3b64454de18fa05c5c484f8909431aa3fabe8b57fe53c1ebbf2eb: Status 404 returned error can't find the container with id 2a889b56b1b3b64454de18fa05c5c484f8909431aa3fabe8b57fe53c1ebbf2eb Oct 30 22:27:56.789740 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:56.789718 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:27:56.791283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:56.791259 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s"] Oct 30 22:27:57.405279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:57.405233 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" event={"ID":"0eb0bc20-8039-48d8-9ab3-6649188a7bf8","Type":"ContainerStarted","Data":"2a889b56b1b3b64454de18fa05c5c484f8909431aa3fabe8b57fe53c1ebbf2eb"} Oct 30 22:27:58.413543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:58.413510 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" event={"ID":"0eb0bc20-8039-48d8-9ab3-6649188a7bf8","Type":"ContainerStarted","Data":"c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5"} Oct 30 22:27:58.413948 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:58.413574 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:27:58.533693 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:27:58.533635 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" podStartSLOduration=2.5564615699999997 podStartE2EDuration="3.533617777s" totalImagesPullingTime="977.156207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:58.532911321 +0000 UTC m=+2457.933467442" watchObservedRunningTime="2026-10-30 22:27:58.533617777 +0000 UTC m=+2457.934173901" Oct 30 22:28:04.425049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:04.425021 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:28:08.995731 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:08.995684 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s"] Oct 30 22:28:08.996256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:08.995934 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" podUID="fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" containerName="maas-api" containerID="cri-o://3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e" gracePeriod=30 Oct 30 22:28:09.293043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.293015 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:28:09.368290 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.368252 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\") pod \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " Oct 30 22:28:09.368551 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.368435 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs\") pod \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " Oct 30 22:28:09.368626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.368549 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca\") pod \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " Oct 30 22:28:09.368626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.368574 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df\" (UniqueName: \"kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df\") pod \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\" (UID: \"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae\") " Oct 30 22:28:09.368923 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.368894 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca" pod "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" (UID: "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:09.369177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.369136 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.371160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.371130 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs" pod "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" (UID: "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:09.371451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.371432 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df" pod "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" (UID: "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae"). InnerVolumeSpecName "kube-api-access-kp4df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:09.371600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.371576 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls" pod "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" (UID: "fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:09.470553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.470527 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.470553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.470552 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp4df\" (UniqueName: \"kubernetes.io/projected/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-kube-api-access-kp4df\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.470727 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.470563 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.480763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.480738 5607 generic.go:356] "Generic (PLEG): container finished" podID="fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" containerID="3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e" exitCode=0 Oct 30 22:28:09.480887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.480785 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" Oct 30 22:28:09.480887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.480835 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" event={"ID":"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae","Type":"ContainerDied","Data":"3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e"} Oct 30 22:28:09.481032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.480929 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s" event={"ID":"fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae","Type":"ContainerDied","Data":"9df69d07ec5cbe7d271579edcaeedce070e6bb60c56a31e6e58fc345b8e847d1"} Oct 30 22:28:09.481032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.480983 5607 scope.go:122] "RemoveContainer" containerID="3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e" Oct 30 22:28:09.494288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.494271 5607 scope.go:122] "RemoveContainer" containerID="3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e" Oct 30 22:28:09.494550 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:09.494533 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e\": container with ID starting with 3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e not found: ID does not exist" containerID="3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e" Oct 30 22:28:09.494606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.494557 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e"} err="failed to get container status \"3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e\": rpc error: code = NotFound desc = could not find container \"3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e\": container with ID starting with 3cf9cea9e750205a61c9d0fc55c17a953467caab19dd4fc25f44b77aad052a9e not found: ID does not exist" Oct 30 22:28:09.650208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.650120 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s"] Oct 30 22:28:09.652480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:09.652454 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ae8879-775fb48478-fk84s"] Oct 30 22:28:10.489874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.489844 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865"] Oct 30 22:28:10.490443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.490428 5607 cpu_manager.go:414] "containerMap: removing container" podUID="fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" containerName="maas-api" Oct 30 22:28:10.490492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.490444 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" containerName="maas-api" Oct 30 22:28:10.490555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.490545 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" containerName="maas-api" Oct 30 22:28:10.495328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.495313 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.548898 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.548879 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1875b4-serving-cert\"" Oct 30 22:28:10.585030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.585005 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.585154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.585057 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.585154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.585117 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.585252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.585153 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8h82\" (UniqueName: \"kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.602458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.602434 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865"] Oct 30 22:28:10.686516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.686482 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.686697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.686527 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.686697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.686560 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.686697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.686588 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8h82\" (UniqueName: \"kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.686697 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:10.686645 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-serving-cert: secret "maas-api-e2e-praxis-a-1875b4-serving-cert" not found Oct 30 22:28:10.686932 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:10.686720 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls podName:d7b8e021-a67e-4d55-b4a7-b5863c65a981 nodeName:}" failed. No retries permitted until 2026-10-30 22:28:11.186700065 +0000 UTC m=+2470.587256179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls") pod "maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" (UID: "d7b8e021-a67e-4d55-b4a7-b5863c65a981") : secret "maas-api-e2e-praxis-a-1875b4-serving-cert" not found Oct 30 22:28:10.687256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.687237 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.689038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.689019 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.751287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.751210 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8h82\" (UniqueName: \"kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:10.870411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:10.870378 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae" path="/var/lib/kubelet/pods/fa55ffc1-b2a9-43ce-93cc-43bfe80d6aae/volumes" Oct 30 22:28:11.191872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:11.191840 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:11.194490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:11.194465 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:11.408689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:11.408665 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:11.791677 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:28:11.791649 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b8e021_a67e_4d55_b4a7_b5863c65a981.slice/crio-732211d528e195b4ced6c597fb37aa71487089c1fe2e72a6b645808324698e57 WatchSource:0}: Error finding container 732211d528e195b4ced6c597fb37aa71487089c1fe2e72a6b645808324698e57: Status 404 returned error can't find the container with id 732211d528e195b4ced6c597fb37aa71487089c1fe2e72a6b645808324698e57 Oct 30 22:28:11.794229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:11.794201 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865"] Oct 30 22:28:12.502404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:12.502367 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" event={"ID":"d7b8e021-a67e-4d55-b4a7-b5863c65a981","Type":"ContainerStarted","Data":"732211d528e195b4ced6c597fb37aa71487089c1fe2e72a6b645808324698e57"} Oct 30 22:28:13.510186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:13.510152 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" event={"ID":"d7b8e021-a67e-4d55-b4a7-b5863c65a981","Type":"ContainerStarted","Data":"1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959"} Oct 30 22:28:13.510632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:13.510370 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:13.633424 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:13.633370 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" podStartSLOduration=2.733906014 podStartE2EDuration="3.633354902s" totalImagesPullingTime="899.448888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:13.631781013 +0000 UTC m=+2473.032337132" watchObservedRunningTime="2026-10-30 22:28:13.633354902 +0000 UTC m=+2473.033911023" Oct 30 22:28:18.672464 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:18.672429 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm"] Oct 30 22:28:18.673056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:18.672674 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" containerID="cri-o://a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a" gracePeriod=30 Oct 30 22:28:19.522548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:19.522516 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:28:22.866412 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:22.866365 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:28:23.939593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:23.939535 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:28:24.036721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.036680 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.036721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.036732 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.036758 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.036807 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.036943 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037015 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037027 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.037268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037081 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037150 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037251 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token\") pod \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\" (UID: \"cf99986a-6113-49bb-bf1e-47dc26f7ddb5\") " Oct 30 22:28:24.037528 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037252 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.037528 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037387 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.037528 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037403 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:24.037864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.037753 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.038186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.038132 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.038283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.038190 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.038283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.038206 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.038283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.038221 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.038283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.038235 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.039341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.039308 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:24.039466 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.039444 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:24.039466 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.039457 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "kube-api-access-l5h6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:24.039588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.039568 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token" pod "cf99986a-6113-49bb-bf1e-47dc26f7ddb5" (UID: "cf99986a-6113-49bb-bf1e-47dc26f7ddb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:24.139147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.139116 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.139147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.139144 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.139147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.139153 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.139475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.139164 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5h6s\" (UniqueName: \"kubernetes.io/projected/cf99986a-6113-49bb-bf1e-47dc26f7ddb5-kube-api-access-l5h6s\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.591131 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.591092 5607 generic.go:356] "Generic (PLEG): container finished" podID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerID="a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a" exitCode=0 Oct 30 22:28:24.591331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.591152 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" Oct 30 22:28:24.591331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.591156 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" event={"ID":"cf99986a-6113-49bb-bf1e-47dc26f7ddb5","Type":"ContainerDied","Data":"a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a"} Oct 30 22:28:24.591331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.591217 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm" event={"ID":"cf99986a-6113-49bb-bf1e-47dc26f7ddb5","Type":"ContainerDied","Data":"cdc6cef39d74b64f78c42ffc5cf414e0155a37fd93ac21139702678929164219"} Oct 30 22:28:24.591331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.591242 5607 scope.go:122] "RemoveContainer" containerID="a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a" Oct 30 22:28:24.602631 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.602612 5607 scope.go:122] "RemoveContainer" containerID="341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4" Oct 30 22:28:24.613260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.613233 5607 scope.go:122] "RemoveContainer" containerID="a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a" Oct 30 22:28:24.613579 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:24.613559 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a\": container with ID starting with a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a not found: ID does not exist" containerID="a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a" Oct 30 22:28:24.613657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.613592 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a"} err="failed to get container status \"a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a\": rpc error: code = NotFound desc = could not find container \"a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a\": container with ID starting with a686c68be9a74731a708f63f288914eaadeb3ea9c84a4ad879b6bf35d6e9502a not found: ID does not exist" Oct 30 22:28:24.613657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.613620 5607 scope.go:122] "RemoveContainer" containerID="341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4" Oct 30 22:28:24.613903 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:24.613886 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4\": container with ID starting with 341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4 not found: ID does not exist" containerID="341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4" Oct 30 22:28:24.613968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.613911 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4"} err="failed to get container status \"341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4\": rpc error: code = NotFound desc = could not find container \"341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4\": container with ID starting with 341d5221a8f4fcedb0bd86ac0d2767d09dd7449e01e72a382e7153b4e84683c4 not found: ID does not exist" Oct 30 22:28:24.761445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.761410 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm"] Oct 30 22:28:24.812351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.812317 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-wkwvm"] Oct 30 22:28:24.870104 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:24.870028 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" path="/var/lib/kubelet/pods/cf99986a-6113-49bb-bf1e-47dc26f7ddb5/volumes" Oct 30 22:28:31.522075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:31.522040 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg"] Oct 30 22:28:31.632457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:31.632426 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6"] Oct 30 22:28:36.522659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.522607 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" podUID="78fc2fef-6c84-437b-a09d-4b6da1170763" containerName="payload-processing" containerID="cri-o://005879c80aba8b3973542502e3a810f6090c7a55145312c516d04819378571b5" gracePeriod=25 Oct 30 22:28:36.634621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.634542 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" podUID="97262a5f-97ba-4fc8-8ee0-02401e8d76ec" containerName="payload-pre-processing" containerID="cri-o://0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c" gracePeriod=25 Oct 30 22:28:36.681471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.681429 5607 generic.go:356] "Generic (PLEG): container finished" podID="78fc2fef-6c84-437b-a09d-4b6da1170763" containerID="005879c80aba8b3973542502e3a810f6090c7a55145312c516d04819378571b5" exitCode=0 Oct 30 22:28:36.681709 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.681678 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" event={"ID":"78fc2fef-6c84-437b-a09d-4b6da1170763","Type":"ContainerDied","Data":"005879c80aba8b3973542502e3a810f6090c7a55145312c516d04819378571b5"} Oct 30 22:28:36.812073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.812045 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:28:36.888798 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.888768 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj\" (UniqueName: \"kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj\") pod \"78fc2fef-6c84-437b-a09d-4b6da1170763\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " Oct 30 22:28:36.888980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.888844 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume\") pod \"78fc2fef-6c84-437b-a09d-4b6da1170763\" (UID: \"78fc2fef-6c84-437b-a09d-4b6da1170763\") " Oct 30 22:28:36.889246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.889216 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume" pod "78fc2fef-6c84-437b-a09d-4b6da1170763" (UID: "78fc2fef-6c84-437b-a09d-4b6da1170763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:36.891230 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.891207 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj" pod "78fc2fef-6c84-437b-a09d-4b6da1170763" (UID: "78fc2fef-6c84-437b-a09d-4b6da1170763"). InnerVolumeSpecName "kube-api-access-zlddj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:36.911805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.911782 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:28:36.990534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.990504 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq\" (UniqueName: \"kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq\") pod \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " Oct 30 22:28:36.990725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.990611 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume\") pod \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\" (UID: \"97262a5f-97ba-4fc8-8ee0-02401e8d76ec\") " Oct 30 22:28:36.991075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.991047 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume" pod "97262a5f-97ba-4fc8-8ee0-02401e8d76ec" (UID: "97262a5f-97ba-4fc8-8ee0-02401e8d76ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:36.993151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.993124 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq" pod "97262a5f-97ba-4fc8-8ee0-02401e8d76ec" (UID: "97262a5f-97ba-4fc8-8ee0-02401e8d76ec"). InnerVolumeSpecName "kube-api-access-5cpmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:36.993536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.993509 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fc2fef-6c84-437b-a09d-4b6da1170763-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:36.993629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.993536 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlddj\" (UniqueName: \"kubernetes.io/projected/78fc2fef-6c84-437b-a09d-4b6da1170763-kube-api-access-zlddj\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:36.993629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:36.993551 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.094666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.094572 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpmq\" (UniqueName: \"kubernetes.io/projected/97262a5f-97ba-4fc8-8ee0-02401e8d76ec-kube-api-access-5cpmq\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.689653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.689619 5607 generic.go:356] "Generic (PLEG): container finished" podID="97262a5f-97ba-4fc8-8ee0-02401e8d76ec" containerID="0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c" exitCode=0 Oct 30 22:28:37.690122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.689667 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" Oct 30 22:28:37.690122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.689712 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" event={"ID":"97262a5f-97ba-4fc8-8ee0-02401e8d76ec","Type":"ContainerDied","Data":"0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c"} Oct 30 22:28:37.690122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.689770 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6" event={"ID":"97262a5f-97ba-4fc8-8ee0-02401e8d76ec","Type":"ContainerDied","Data":"dc1e848148e60e49dc256ecbb68cdd4e789c413a2136095518d78ac566850e1c"} Oct 30 22:28:37.690122 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.689792 5607 scope.go:122] "RemoveContainer" containerID="0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c" Oct 30 22:28:37.691394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.691369 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" event={"ID":"78fc2fef-6c84-437b-a09d-4b6da1170763","Type":"ContainerDied","Data":"a4fde8d78013f308bc883a0055cd747301cb18142f8e374778c8348dc3c32bde"} Oct 30 22:28:37.691472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.691381 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg" Oct 30 22:28:37.701076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.701056 5607 scope.go:122] "RemoveContainer" containerID="0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c" Oct 30 22:28:37.701541 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:37.701403 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c\": container with ID starting with 0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c not found: ID does not exist" containerID="0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c" Oct 30 22:28:37.701541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.701444 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c"} err="failed to get container status \"0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c\": rpc error: code = NotFound desc = could not find container \"0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c\": container with ID starting with 0549fb5423306fd10754256b4992b73f1e7656b4aaa2e14f3a0f44f670423a1c not found: ID does not exist" Oct 30 22:28:37.701541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.701468 5607 scope.go:122] "RemoveContainer" containerID="005879c80aba8b3973542502e3a810f6090c7a55145312c516d04819378571b5" Oct 30 22:28:37.858262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.858185 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6"] Oct 30 22:28:37.909062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.909036 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf5019-79f6d98479-pm5v6"] Oct 30 22:28:37.974697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.974662 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg"] Oct 30 22:28:37.979779 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:37.979754 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cf5019-6bd7498b77-72hjg"] Oct 30 22:28:38.870862 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:38.870827 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fc2fef-6c84-437b-a09d-4b6da1170763" path="/var/lib/kubelet/pods/78fc2fef-6c84-437b-a09d-4b6da1170763/volumes" Oct 30 22:28:38.871281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:38.871261 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97262a5f-97ba-4fc8-8ee0-02401e8d76ec" path="/var/lib/kubelet/pods/97262a5f-97ba-4fc8-8ee0-02401e8d76ec/volumes" Oct 30 22:28:46.920856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.920817 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w"] Oct 30 22:28:46.921724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921695 5607 cpu_manager.go:414] "containerMap: removing container" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" Oct 30 22:28:46.921724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921724 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" Oct 30 22:28:46.921976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921760 5607 cpu_manager.go:414] "containerMap: removing container" podUID="97262a5f-97ba-4fc8-8ee0-02401e8d76ec" containerName="payload-pre-processing" Oct 30 22:28:46.921976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921769 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97262a5f-97ba-4fc8-8ee0-02401e8d76ec" containerName="payload-pre-processing" Oct 30 22:28:46.921976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921805 5607 cpu_manager.go:414] "containerMap: removing container" podUID="78fc2fef-6c84-437b-a09d-4b6da1170763" containerName="payload-processing" Oct 30 22:28:46.921976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921813 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fc2fef-6c84-437b-a09d-4b6da1170763" containerName="payload-processing" Oct 30 22:28:46.921976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921834 5607 cpu_manager.go:414] "containerMap: removing container" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" Oct 30 22:28:46.921976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.921842 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" Oct 30 22:28:46.922355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.922009 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="97262a5f-97ba-4fc8-8ee0-02401e8d76ec" containerName="payload-pre-processing" Oct 30 22:28:46.922355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.922032 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fc2fef-6c84-437b-a09d-4b6da1170763" containerName="payload-processing" Oct 30 22:28:46.922355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.922043 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" Oct 30 22:28:46.922355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.922058 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf99986a-6113-49bb-bf1e-47dc26f7ddb5" containerName="istio-proxy" Oct 30 22:28:46.927656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.927633 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:46.990985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:46.985325 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c301cf-openshift-default-dockercfg-gph96\"" Oct 30 22:28:47.040692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.040658 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w"] Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109448 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109520 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109578 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109651 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109684 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109717 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109760 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvz6\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109822 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.111001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.109925 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.160009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.159975 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z"] Oct 30 22:28:47.164287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.164262 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.214710 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214671 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.214897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214735 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.214897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214787 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.214897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214828 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.214897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214854 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.214897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214883 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.215255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.214920 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvz6\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.215255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.215015 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.215255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.215079 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.215446 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.215293 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.215573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.215548 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.215833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.215806 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.216861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.216812 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.217456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.217431 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.218607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.218547 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.234614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.219198 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.290327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.290288 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.290495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.290429 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvz6\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6\") pod \"e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.292902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.292873 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z"] Oct 30 22:28:47.316693 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.316646 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.316846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.316733 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.316846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.316772 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.316846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.316822 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.317177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.316854 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.317177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.316902 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.317177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.317121 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.317177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.317155 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.317500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.317211 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdff7\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.418704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.418664 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.418937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.418729 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.418937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.418778 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.418937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.418806 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.418937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.418850 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.419299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.418943 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.419299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.419004 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.419299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.419056 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdff7\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.419299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.419172 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.420396 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.421039 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.421224 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.421548 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.422214 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.424695 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.429757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.429735 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.489007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.488973 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.489142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.489072 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdff7\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.547389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.547358 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:47.778088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.778049 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:47.974002 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:28:47.973970 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2064e2f8_2156_47bc_9601_82874abb8df7.slice/crio-30fdf39c97ce3a1f879eeff5697f67ac0343e83c67475f47239c09d5a6fe16e3 WatchSource:0}: Error finding container 30fdf39c97ce3a1f879eeff5697f67ac0343e83c67475f47239c09d5a6fe16e3: Status 404 returned error can't find the container with id 30fdf39c97ce3a1f879eeff5697f67ac0343e83c67475f47239c09d5a6fe16e3 Oct 30 22:28:47.976469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.976436 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:47.976598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.976506 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:47.976598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.976558 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:47.977882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:47.977857 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w"] Oct 30 22:28:48.157264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.157229 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z"] Oct 30 22:28:48.157513 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.157481 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:48.157606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.157560 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:48.157671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.157604 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:28:48.766031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.765985 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" event={"ID":"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d","Type":"ContainerStarted","Data":"0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc"} Oct 30 22:28:48.766240 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.766053 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" event={"ID":"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d","Type":"ContainerStarted","Data":"789c28f0f604021531422c605903d3b03292d05982b83f7c2a855f529ad64b9e"} Oct 30 22:28:48.767624 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.767592 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" event={"ID":"2064e2f8-2156-47bc-9601-82874abb8df7","Type":"ContainerStarted","Data":"7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37"} Oct 30 22:28:48.767763 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.767638 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" event={"ID":"2064e2f8-2156-47bc-9601-82874abb8df7","Type":"ContainerStarted","Data":"30fdf39c97ce3a1f879eeff5697f67ac0343e83c67475f47239c09d5a6fe16e3"} Oct 30 22:28:48.778274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.778254 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:48.891909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:48.891858 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podStartSLOduration=1.8918432570000001 podStartE2EDuration="1.891843257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:48.890503108 +0000 UTC m=+2508.291059229" watchObservedRunningTime="2026-10-30 22:28:48.891843257 +0000 UTC m=+2508.292399378" Oct 30 22:28:49.020041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.019377 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podStartSLOduration=3.019356085 podStartE2EDuration="3.019356085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:49.016493774 +0000 UTC m=+2508.417049897" watchObservedRunningTime="2026-10-30 22:28:49.019356085 +0000 UTC m=+2508.419912211" Oct 30 22:28:49.551999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.551508 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:49.559004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.558977 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:49.773555 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.773521 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:49.774415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.774393 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:28:49.780529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.780504 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:28:49.780637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:49.780553 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:28:50.778418 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:50.778379 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:28:50.778927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:50.778438 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:28:51.778906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:51.778873 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:28:51.779345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:51.778931 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:28:52.778726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:52.778689 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:28:52.778998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:52.778747 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:28:52.864688 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:28:52.864647 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:28:53.778474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:53.778423 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:28:53.778698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:53.778485 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:28:54.703224 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:54.703186 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx"] Oct 30 22:28:54.704502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:54.704467 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" containerID="cri-o://c18c7b64106f71f1b6a573e4a1bbe49a24ab14e47eeec6316dc1b6a817c38154" gracePeriod=30 Oct 30 22:28:54.779322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:54.779290 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:28:54.779488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:54.779355 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:28:55.611782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.611753 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp"] Oct 30 22:28:55.615711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.615695 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.671813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.671790 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c301cf\"" Oct 30 22:28:55.672034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.672011 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c301cf-dockercfg-fzvqb\"" Oct 30 22:28:55.675617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.675596 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp"] Oct 30 22:28:55.742506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.742470 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9fqz\" (UniqueName: \"kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz\") pod \"payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.743098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.742528 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.768231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.768099 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx"] Oct 30 22:28:55.772492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.772466 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:55.784893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.784864 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:55.785612 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.785591 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:55.786393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.786340 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:28:55.827895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.827869 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx"] Oct 30 22:28:55.843794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.843766 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9fqz\" (UniqueName: \"kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz\") pod \"payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.844003 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.843809 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.844003 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.843912 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p678\" (UniqueName: \"kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678\") pod \"payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:55.844183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.844014 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:55.844702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.844675 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.906925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.906863 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9fqz\" (UniqueName: \"kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz\") pod \"payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.930654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.930628 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:55.948443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.947283 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p678\" (UniqueName: \"kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678\") pod \"payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:55.948443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.947363 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:55.949164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:55.949138 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:56.026572 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.026542 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p678\" (UniqueName: \"kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678\") pod \"payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:56.091133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.091103 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:56.310009 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:28:56.309970 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7f4ee1_1516_4cf6_a581_72db7e851a69.slice/crio-7051ecf4ea57550588c7c2c5d544df0f19cf24be620c297906f4f433c2aef037 WatchSource:0}: Error finding container 7051ecf4ea57550588c7c2c5d544df0f19cf24be620c297906f4f433c2aef037: Status 404 returned error can't find the container with id 7051ecf4ea57550588c7c2c5d544df0f19cf24be620c297906f4f433c2aef037 Oct 30 22:28:56.312105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.312083 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp"] Oct 30 22:28:56.462618 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:28:56.462592 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dcd691_fe9b_4b17_addc_504be4367090.slice/crio-d316f0fbdbaf4c48a395578b24db3c47704a218339580632b70003e2d68efb05 WatchSource:0}: Error finding container d316f0fbdbaf4c48a395578b24db3c47704a218339580632b70003e2d68efb05: Status 404 returned error can't find the container with id d316f0fbdbaf4c48a395578b24db3c47704a218339580632b70003e2d68efb05 Oct 30 22:28:56.466094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.466069 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx"] Oct 30 22:28:56.836578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.836541 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" event={"ID":"a7dcd691-fe9b-4b17-addc-504be4367090","Type":"ContainerStarted","Data":"8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552"} Oct 30 22:28:56.837081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.836601 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" event={"ID":"a7dcd691-fe9b-4b17-addc-504be4367090","Type":"ContainerStarted","Data":"d316f0fbdbaf4c48a395578b24db3c47704a218339580632b70003e2d68efb05"} Oct 30 22:28:56.837081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.836636 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:28:56.838211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.838185 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" event={"ID":"bc7f4ee1-1516-4cf6-a581-72db7e851a69","Type":"ContainerStarted","Data":"cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287"} Oct 30 22:28:56.838327 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.838228 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" event={"ID":"bc7f4ee1-1516-4cf6-a581-72db7e851a69","Type":"ContainerStarted","Data":"7051ecf4ea57550588c7c2c5d544df0f19cf24be620c297906f4f433c2aef037"} Oct 30 22:28:56.958593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:56.958543 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" podStartSLOduration=1.95852936 podStartE2EDuration="1.95852936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:56.956076491 +0000 UTC m=+2516.356632624" watchObservedRunningTime="2026-10-30 22:28:56.95852936 +0000 UTC m=+2516.359085480" Oct 30 22:28:57.078996 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:57.078893 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" podStartSLOduration=2.078872239 podStartE2EDuration="2.078872239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:57.075868011 +0000 UTC m=+2516.476424134" watchObservedRunningTime="2026-10-30 22:28:57.078872239 +0000 UTC m=+2516.479428360" Oct 30 22:28:57.843839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:57.843800 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:28:59.860389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:59.860354 5607 generic.go:356] "Generic (PLEG): container finished" podID="4542fc41-1931-4104-b621-086e14239149" containerID="c18c7b64106f71f1b6a573e4a1bbe49a24ab14e47eeec6316dc1b6a817c38154" exitCode=0 Oct 30 22:28:59.860810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:59.860440 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" event={"ID":"4542fc41-1931-4104-b621-086e14239149","Type":"ContainerDied","Data":"c18c7b64106f71f1b6a573e4a1bbe49a24ab14e47eeec6316dc1b6a817c38154"} Oct 30 22:28:59.991511 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:28:59.991440 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 22:29:00.098505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098474 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.098679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098545 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.098679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098614 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.098679 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098654 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.098841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098681 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.098841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098772 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.098841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098806 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.099052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098858 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:00.099052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098869 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.099052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098986 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:00.099052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.098991 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token\") pod \"4542fc41-1931-4104-b621-086e14239149\" (UID: \"4542fc41-1931-4104-b621-086e14239149\") " Oct 30 22:29:00.099246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099142 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:00.099246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099106 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:00.099451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099427 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:00.099609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099588 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.099651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099616 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4542fc41-1931-4104-b621-086e14239149-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.099651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099633 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.099651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099648 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.099776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.099663 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.101099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.101071 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:00.101099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.101076 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "kube-api-access-rflqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:00.101373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.101349 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:00.101594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.101574 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy" pod "4542fc41-1931-4104-b621-086e14239149" (UID: "4542fc41-1931-4104-b621-086e14239149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:00.200687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.200661 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflqb\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-kube-api-access-rflqb\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.200687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.200688 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4542fc41-1931-4104-b621-086e14239149-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.200865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.200699 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4542fc41-1931-4104-b621-086e14239149-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.200865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.200708 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4542fc41-1931-4104-b621-086e14239149-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:00.870997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.870968 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" Oct 30 22:29:00.872695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.872666 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx" event={"ID":"4542fc41-1931-4104-b621-086e14239149","Type":"ContainerDied","Data":"c456f9cc9e1eddcae29d0c5d71b4b3acb166011f9df414b5543b78ad6644438f"} Oct 30 22:29:00.872849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:00.872719 5607 scope.go:122] "RemoveContainer" containerID="c18c7b64106f71f1b6a573e4a1bbe49a24ab14e47eeec6316dc1b6a817c38154" Oct 30 22:29:01.048235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:01.044923 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx"] Oct 30 22:29:01.092500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:01.092471 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-gqbnx"] Oct 30 22:29:02.870349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:02.870317 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4542fc41-1931-4104-b621-086e14239149" path="/var/lib/kubelet/pods/4542fc41-1931-4104-b621-086e14239149/volumes" Oct 30 22:29:07.846447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:07.846419 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:29:08.854895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:08.854866 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:29:13.427090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.427003 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4"] Oct 30 22:29:13.427846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.427823 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" Oct 30 22:29:13.428016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.427848 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" Oct 30 22:29:13.428123 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.428108 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4542fc41-1931-4104-b621-086e14239149" containerName="istio-proxy" Oct 30 22:29:13.433413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.433390 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481393 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481500 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481539 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481558 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481579 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481774 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481619 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481780 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmb5\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481812 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.481840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.481828 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.541074 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.541040 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b47e8d-openshift-default-dockercfg-bxx6t\"" Oct 30 22:29:13.543781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.543751 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4"] Oct 30 22:29:13.582715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.582684 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.582874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.582752 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.582874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.582837 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmb5\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.582879 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.582897 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.582919 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.583032 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.583078 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.583105 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.583275 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.583349 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.583768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.583744 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.584051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.584033 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.584136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.584090 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.585459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.585441 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.585768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.585747 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.644930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.644906 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.645161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.645143 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmb5\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:13.747316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:13.747293 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:14.179707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:14.179659 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:14.179831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:14.179755 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:14.179831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:14.179798 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:14.181457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:14.181435 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4"] Oct 30 22:29:14.960506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:14.960453 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" event={"ID":"af51ad9f-7f7a-4b81-9fcc-6b507843e45a","Type":"ContainerStarted","Data":"f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818"} Oct 30 22:29:14.961023 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:14.960520 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" event={"ID":"af51ad9f-7f7a-4b81-9fcc-6b507843e45a","Type":"ContainerStarted","Data":"0840d9f060b5b30d99a61a53519b76a223adc7e8187e00bbd338a290d1719fe3"} Oct 30 22:29:15.752061 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:15.752024 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:15.754979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:15.753732 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:29:15.754979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:15.753791 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:29:16.748165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:16.748130 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:29:16.748567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:16.748182 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:29:17.747990 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:17.747940 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:29:17.748209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:17.748011 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:29:18.748052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:18.748012 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:29:18.748250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:18.748101 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:29:19.749444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:19.748851 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:29:19.749444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:19.748925 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:29:20.748221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:20.748188 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 22:29:20.748483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:20.748235 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 22:29:21.752680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:21.752642 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:21.753190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:21.753127 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:21.753999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:21.753981 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:29:21.931043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:21.930972 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podStartSLOduration=8.930936272 podStartE2EDuration="8.930936272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:15.116519532 +0000 UTC m=+2534.517075653" watchObservedRunningTime="2026-10-30 22:29:21.930936272 +0000 UTC m=+2541.331492394" Oct 30 22:29:22.863760 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:29:22.863727 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:29:47.633434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.633399 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb"] Oct 30 22:29:47.638985 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.638932 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.731841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.731806 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.731983 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732061 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732112 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732163 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732184 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktpnb\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732221 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732243 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.732490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.732285 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833574 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833620 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833667 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833721 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833844 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktpnb\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833912 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.833991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.833941 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.834165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.834039 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.834165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.834111 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.834165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.834147 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.834320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.834203 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.834463 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.834439 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.834519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.834459 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.835078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.835053 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.836458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.836427 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.836702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.836683 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.848356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.848327 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb"] Oct 30 22:29:47.896838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.896771 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.897099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.897075 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktpnb\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:47.953637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:47.953609 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:48.378670 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:29:48.378641 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9120d792_d37d_4255_82df_8cf3be1fcff2.slice/crio-e74f3166bf8225c6864408c0891aba8260b98b3566568a98ac89760379475ebe WatchSource:0}: Error finding container e74f3166bf8225c6864408c0891aba8260b98b3566568a98ac89760379475ebe: Status 404 returned error can't find the container with id e74f3166bf8225c6864408c0891aba8260b98b3566568a98ac89760379475ebe Oct 30 22:29:48.381095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:48.381061 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:48.381227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:48.381138 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb"] Oct 30 22:29:48.381227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:48.381143 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:48.381365 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:48.381251 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:49.215978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:49.215881 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" event={"ID":"9120d792-d37d-4255-82df-8cf3be1fcff2","Type":"ContainerStarted","Data":"ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8"} Oct 30 22:29:49.215978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:49.215967 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" event={"ID":"9120d792-d37d-4255-82df-8cf3be1fcff2","Type":"ContainerStarted","Data":"e74f3166bf8225c6864408c0891aba8260b98b3566568a98ac89760379475ebe"} Oct 30 22:29:49.391308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:49.391254 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podStartSLOduration=2.391240457 podStartE2EDuration="2.391240457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:49.389803393 +0000 UTC m=+2568.790359514" watchObservedRunningTime="2026-10-30 22:29:49.391240457 +0000 UTC m=+2568.791796578" Oct 30 22:29:49.953719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:49.953687 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:49.955548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:49.955524 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 22:29:49.955686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:49.955569 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 22:29:50.954591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:50.954555 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 22:29:50.955022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:50.954607 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 22:29:51.954861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:51.954823 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 22:29:51.955305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:51.954892 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 22:29:52.865604 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:29:52.865565 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:29:52.954073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:52.954037 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 22:29:52.954246 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:52.954093 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 22:29:53.954240 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:53.954204 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 22:29:53.954694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:53.954254 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 22:29:54.954204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:54.954169 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 22:29:54.954391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:54.954227 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 22:29:55.958772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:55.958740 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:55.959294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:55.959149 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:29:55.959717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:29:55.959697 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:30:00.185394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.185343 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g"] Oct 30 22:30:00.190511 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.190487 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch"] Oct 30 22:30:00.190651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.190592 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.195163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.195140 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.224661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.224632 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6"] Oct 30 22:30:00.228925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.228907 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.236076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.236057 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8"] Oct 30 22:30:00.240211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.240187 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.277789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.277766 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g"] Oct 30 22:30:00.282095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.282064 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch"] Oct 30 22:30:00.301682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.301641 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:30:00.302049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.302026 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8"] Oct 30 22:30:00.302999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.302973 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ckn\" (UniqueName: \"kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn\") pod \"maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.303243 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.303216 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.303409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.303371 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.303518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.303490 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmk84\" (UniqueName: \"kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.348838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.348804 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6"] Oct 30 22:30:00.405183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405156 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.405367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405192 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.405367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405247 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.405367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405332 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.405548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405415 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmk84\" (UniqueName: \"kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.405548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405522 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs575\" (UniqueName: \"kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575\") pod \"maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.405657 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405603 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ckn\" (UniqueName: \"kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn\") pod \"maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.405724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405695 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9tg\" (UniqueName: \"kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg\") pod \"maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.405901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.405886 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.406052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.406035 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.468282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.468220 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmk84\" (UniqueName: \"kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.468282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.468220 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ckn\" (UniqueName: \"kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn\") pod \"maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.506573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.506544 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9tg\" (UniqueName: \"kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg\") pod \"maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.506726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.506593 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.506726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.506654 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.506843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.506751 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs575\" (UniqueName: \"kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575\") pod \"maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.506843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.506820 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:00.507341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.507317 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.507450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.507377 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.515120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.515099 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:00.573302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.571147 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs575\" (UniqueName: \"kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575\") pod \"maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.573302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.571937 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9tg\" (UniqueName: \"kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg\") pod \"maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.844890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.844820 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:00.857092 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.857067 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:00.888503 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:00.888471 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dafc1a_fe49_4b34_95f0_c8d5f39a6d6f.slice/crio-c6def12f14fb7ce2c0f18a182f9cd202c9ef5f07efc422822608e9bdcc1c5c6a WatchSource:0}: Error finding container c6def12f14fb7ce2c0f18a182f9cd202c9ef5f07efc422822608e9bdcc1c5c6a: Status 404 returned error can't find the container with id c6def12f14fb7ce2c0f18a182f9cd202c9ef5f07efc422822608e9bdcc1c5c6a Oct 30 22:30:00.891851 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.891822 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g"] Oct 30 22:30:00.943249 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:00.943215 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b8223d_c52c_4947_b368_0b85c9438806.slice/crio-597684d2ce078d9618e3f7b801fbd094df6197b85b948d8901331094a12f22a9 WatchSource:0}: Error finding container 597684d2ce078d9618e3f7b801fbd094df6197b85b948d8901331094a12f22a9: Status 404 returned error can't find the container with id 597684d2ce078d9618e3f7b801fbd094df6197b85b948d8901331094a12f22a9 Oct 30 22:30:00.947153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:00.944859 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch"] Oct 30 22:30:01.226569 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:01.226151 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a37637c_f2e6_477a_904f_95c217afa0d7.slice/crio-eb73854502c4853adba14bb36a80c2cb8bc913f808b06fdbb5b49b1d0d9d98ce WatchSource:0}: Error finding container eb73854502c4853adba14bb36a80c2cb8bc913f808b06fdbb5b49b1d0d9d98ce: Status 404 returned error can't find the container with id eb73854502c4853adba14bb36a80c2cb8bc913f808b06fdbb5b49b1d0d9d98ce Oct 30 22:30:01.229713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.229689 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6"] Oct 30 22:30:01.256519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.256500 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8"] Oct 30 22:30:01.260119 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:01.260085 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd191d4b0_7e8c_4d71_907a_ad533bd96ff0.slice/crio-cd6f550bbb5c651b588685070b2e7792bcb9d304b89b4777287ed35271d4c8b0 WatchSource:0}: Error finding container cd6f550bbb5c651b588685070b2e7792bcb9d304b89b4777287ed35271d4c8b0: Status 404 returned error can't find the container with id cd6f550bbb5c651b588685070b2e7792bcb9d304b89b4777287ed35271d4c8b0 Oct 30 22:30:01.305864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.305830 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerStarted","Data":"cd6f550bbb5c651b588685070b2e7792bcb9d304b89b4777287ed35271d4c8b0"} Oct 30 22:30:01.307354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.307327 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" event={"ID":"6a37637c-f2e6-477a-904f-95c217afa0d7","Type":"ContainerStarted","Data":"eb73854502c4853adba14bb36a80c2cb8bc913f808b06fdbb5b49b1d0d9d98ce"} Oct 30 22:30:01.308968 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.308920 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerStarted","Data":"597684d2ce078d9618e3f7b801fbd094df6197b85b948d8901331094a12f22a9"} Oct 30 22:30:01.310495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.310475 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerStarted","Data":"6b1879429d39d174da904893a52ca1cc12e9fa4a798b4696977b13bfe3aba6ef"} Oct 30 22:30:01.310590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.310517 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerStarted","Data":"c6def12f14fb7ce2c0f18a182f9cd202c9ef5f07efc422822608e9bdcc1c5c6a"} Oct 30 22:30:01.435051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:01.435009 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" podStartSLOduration=1.434995782 podStartE2EDuration="1.434995782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.43319266 +0000 UTC m=+2580.833748782" watchObservedRunningTime="2026-10-30 22:30:01.434995782 +0000 UTC m=+2580.835551902" Oct 30 22:30:02.318815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.318781 5607 generic.go:356] "Generic (PLEG): container finished" podID="c8b8223d-c52c-4947-b368-0b85c9438806" containerID="2c2329e1ce34c0909b677e9ea55320fb4a906c1fd217660619f665241ade3d10" exitCode=60 Oct 30 22:30:02.319257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.318878 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerDied","Data":"2c2329e1ce34c0909b677e9ea55320fb4a906c1fd217660619f665241ade3d10"} Oct 30 22:30:02.319257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.319173 5607 scope.go:122] "RemoveContainer" containerID="2c2329e1ce34c0909b677e9ea55320fb4a906c1fd217660619f665241ade3d10" Oct 30 22:30:02.320606 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.320584 5607 generic.go:356] "Generic (PLEG): container finished" podID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerID="6b1879429d39d174da904893a52ca1cc12e9fa4a798b4696977b13bfe3aba6ef" exitCode=60 Oct 30 22:30:02.320716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.320677 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerDied","Data":"6b1879429d39d174da904893a52ca1cc12e9fa4a798b4696977b13bfe3aba6ef"} Oct 30 22:30:02.320917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.320904 5607 scope.go:122] "RemoveContainer" containerID="6b1879429d39d174da904893a52ca1cc12e9fa4a798b4696977b13bfe3aba6ef" Oct 30 22:30:02.322486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.322361 5607 generic.go:356] "Generic (PLEG): container finished" podID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerID="528c51638c9165990d5ba17d79207f8a3a0b5f7d7d40b10478e9ca53c6980c50" exitCode=60 Oct 30 22:30:02.322486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.322449 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerDied","Data":"528c51638c9165990d5ba17d79207f8a3a0b5f7d7d40b10478e9ca53c6980c50"} Oct 30 22:30:02.322733 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.322631 5607 scope.go:122] "RemoveContainer" containerID="528c51638c9165990d5ba17d79207f8a3a0b5f7d7d40b10478e9ca53c6980c50" Oct 30 22:30:02.324376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.324356 5607 generic.go:356] "Generic (PLEG): container finished" podID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerID="a24ea48c3b5371794233965313790a526a5d3015da8f8f13bcfc93470fe7ce57" exitCode=7 Oct 30 22:30:02.324465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.324407 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" event={"ID":"6a37637c-f2e6-477a-904f-95c217afa0d7","Type":"ContainerDied","Data":"a24ea48c3b5371794233965313790a526a5d3015da8f8f13bcfc93470fe7ce57"} Oct 30 22:30:02.324660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:02.324623 5607 scope.go:122] "RemoveContainer" containerID="a24ea48c3b5371794233965313790a526a5d3015da8f8f13bcfc93470fe7ce57" Oct 30 22:30:03.334119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.334089 5607 generic.go:356] "Generic (PLEG): container finished" podID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerID="9e31e87dc4f576dae2f46eee2c6ab71b156937161e3be4f2cd1c1dcacad18461" exitCode=60 Oct 30 22:30:03.334519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.334177 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" event={"ID":"6a37637c-f2e6-477a-904f-95c217afa0d7","Type":"ContainerDied","Data":"9e31e87dc4f576dae2f46eee2c6ab71b156937161e3be4f2cd1c1dcacad18461"} Oct 30 22:30:03.334519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.334252 5607 scope.go:122] "RemoveContainer" containerID="a24ea48c3b5371794233965313790a526a5d3015da8f8f13bcfc93470fe7ce57" Oct 30 22:30:03.334628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.334563 5607 scope.go:122] "RemoveContainer" containerID="9e31e87dc4f576dae2f46eee2c6ab71b156937161e3be4f2cd1c1dcacad18461" Oct 30 22:30:03.334914 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:03.334894 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6_odh-ai-gateway-infra(6a37637c-f2e6-477a-904f-95c217afa0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" Oct 30 22:30:03.336340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.336305 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerStarted","Data":"00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3"} Oct 30 22:30:03.338332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.338308 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerStarted","Data":"8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057"} Oct 30 22:30:03.340554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.340524 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerStarted","Data":"7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb"} Oct 30 22:30:03.572237 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.572152 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" podStartSLOduration=3.572139012 podStartE2EDuration="3.572139012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.571217614 +0000 UTC m=+2582.971773737" watchObservedRunningTime="2026-10-30 22:30:03.572139012 +0000 UTC m=+2582.972695132" Oct 30 22:30:03.815016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:03.814969 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" podStartSLOduration=3.814944981 podStartE2EDuration="3.814944981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.812775851 +0000 UTC m=+2583.213331971" watchObservedRunningTime="2026-10-30 22:30:03.814944981 +0000 UTC m=+2583.215501101" Oct 30 22:30:04.349660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.349623 5607 generic.go:356] "Generic (PLEG): container finished" podID="c8b8223d-c52c-4947-b368-0b85c9438806" containerID="00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3" exitCode=60 Oct 30 22:30:04.350175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.349707 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerDied","Data":"00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3"} Oct 30 22:30:04.350175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.349779 5607 scope.go:122] "RemoveContainer" containerID="2c2329e1ce34c0909b677e9ea55320fb4a906c1fd217660619f665241ade3d10" Oct 30 22:30:04.350175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.350044 5607 scope.go:122] "RemoveContainer" containerID="00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3" Oct 30 22:30:04.350415 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:04.350394 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch_odh-ai-gateway-infra(c8b8223d-c52c-4947-b368-0b85c9438806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" Oct 30 22:30:04.352021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.352002 5607 generic.go:356] "Generic (PLEG): container finished" podID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerID="8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057" exitCode=60 Oct 30 22:30:04.352119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.352084 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerDied","Data":"8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057"} Oct 30 22:30:04.352364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.352346 5607 scope.go:122] "RemoveContainer" containerID="8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057" Oct 30 22:30:04.352689 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:04.352668 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g_odh-ai-gateway-infra(91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" Oct 30 22:30:04.353934 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.353913 5607 generic.go:356] "Generic (PLEG): container finished" podID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerID="7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb" exitCode=60 Oct 30 22:30:04.354064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.354026 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerDied","Data":"7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb"} Oct 30 22:30:04.354265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.354249 5607 scope.go:122] "RemoveContainer" containerID="7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb" Oct 30 22:30:04.354593 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:04.354546 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8_odh-ai-gateway-infra(d191d4b0-7e8c-4d71-907a-ad533bd96ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" Oct 30 22:30:04.355932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.355912 5607 scope.go:122] "RemoveContainer" containerID="9e31e87dc4f576dae2f46eee2c6ab71b156937161e3be4f2cd1c1dcacad18461" Oct 30 22:30:04.356277 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:04.356252 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6_odh-ai-gateway-infra(6a37637c-f2e6-477a-904f-95c217afa0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" Oct 30 22:30:04.363116 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.363101 5607 scope.go:122] "RemoveContainer" containerID="6b1879429d39d174da904893a52ca1cc12e9fa4a798b4696977b13bfe3aba6ef" Oct 30 22:30:04.375856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.375828 5607 scope.go:122] "RemoveContainer" containerID="528c51638c9165990d5ba17d79207f8a3a0b5f7d7d40b10478e9ca53c6980c50" Oct 30 22:30:04.727650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.727608 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p"] Oct 30 22:30:04.734370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.734348 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:04.782831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.782809 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p"] Oct 30 22:30:04.788212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.788188 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9717a8\"" Oct 30 22:30:04.788353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.788317 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9717a8-dockercfg-m2xqb\"" Oct 30 22:30:04.865859 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.865830 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72g9\" (UniqueName: \"kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9\") pod \"payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:04.866049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.866005 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:04.967492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.967458 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72g9\" (UniqueName: \"kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9\") pod \"payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:04.967688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.967616 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:04.968518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:04.968494 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:05.030865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:05.030811 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72g9\" (UniqueName: \"kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9\") pod \"payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:05.049857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:05.049836 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:05.363603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:05.363519 5607 scope.go:122] "RemoveContainer" containerID="00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3" Oct 30 22:30:05.364055 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:05.363888 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch_odh-ai-gateway-infra(c8b8223d-c52c-4947-b368-0b85c9438806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" Oct 30 22:30:05.365141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:05.365119 5607 scope.go:122] "RemoveContainer" containerID="8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057" Oct 30 22:30:05.365409 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:05.365389 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g_odh-ai-gateway-infra(91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" Oct 30 22:30:05.366615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:05.366598 5607 scope.go:122] "RemoveContainer" containerID="7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb" Oct 30 22:30:05.366807 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:05.366791 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8_odh-ai-gateway-infra(d191d4b0-7e8c-4d71-907a-ad533bd96ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" Oct 30 22:30:05.421082 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:05.421056 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45326de7_55f6_4593_9ae2_827a3f87eefb.slice/crio-6f52827129ee45a07c3a048a9f1a5de2b5e900c05ec04b34b90618f928d00eca WatchSource:0}: Error finding container 6f52827129ee45a07c3a048a9f1a5de2b5e900c05ec04b34b90618f928d00eca: Status 404 returned error can't find the container with id 6f52827129ee45a07c3a048a9f1a5de2b5e900c05ec04b34b90618f928d00eca Oct 30 22:30:05.423742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:05.423723 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p"] Oct 30 22:30:06.374179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:06.374137 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" event={"ID":"45326de7-55f6-4593-9ae2-827a3f87eefb","Type":"ContainerStarted","Data":"11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192"} Oct 30 22:30:06.374580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:06.374200 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" event={"ID":"45326de7-55f6-4593-9ae2-827a3f87eefb","Type":"ContainerStarted","Data":"6f52827129ee45a07c3a048a9f1a5de2b5e900c05ec04b34b90618f928d00eca"} Oct 30 22:30:06.374580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:06.374333 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:06.495146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:06.495100 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" podStartSLOduration=2.495086288 podStartE2EDuration="2.495086288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:06.493140763 +0000 UTC m=+2585.893696884" watchObservedRunningTime="2026-10-30 22:30:06.495086288 +0000 UTC m=+2585.895642409" Oct 30 22:30:13.891703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:13.891669 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l"] Oct 30 22:30:13.895633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:13.895615 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.003387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.003359 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l"] Oct 30 22:30:14.033738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.033708 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.033903 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.033770 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.033903 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.033825 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2rp\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.034030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.033896 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.034030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.033981 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.034105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.034036 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.034105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.034092 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.034180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.034149 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.034213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.034192 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135305 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135341 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135368 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135420 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135467 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135508 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2rp\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135536 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135608 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135630 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135743 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135696 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.135945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135790 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.136048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.135974 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.136319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.136292 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.136443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.136422 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.137916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.137891 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.138047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.138032 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.221547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.221516 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2rp\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.222067 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.222042 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.509001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.508898 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:14.863116 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.863027 5607 scope.go:122] "RemoveContainer" containerID="7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb" Oct 30 22:30:14.985035 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:14.985008 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c37f3f_7ae1_441a_80f6_a7e5e245dfc0.slice/crio-c9c2a0d0dd573f88760ea977dfaeddd14400c1d95e5cb33a815d16ef12d97c84 WatchSource:0}: Error finding container c9c2a0d0dd573f88760ea977dfaeddd14400c1d95e5cb33a815d16ef12d97c84: Status 404 returned error can't find the container with id c9c2a0d0dd573f88760ea977dfaeddd14400c1d95e5cb33a815d16ef12d97c84 Oct 30 22:30:14.987621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.987584 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:14.987687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.987674 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:14.987754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.987739 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:14.988737 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:14.988713 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l"] Oct 30 22:30:15.480870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:15.480818 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" event={"ID":"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0","Type":"ContainerStarted","Data":"7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad"} Oct 30 22:30:15.481077 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:15.480903 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" event={"ID":"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0","Type":"ContainerStarted","Data":"c9c2a0d0dd573f88760ea977dfaeddd14400c1d95e5cb33a815d16ef12d97c84"} Oct 30 22:30:15.482927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:15.482903 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerStarted","Data":"a76ceeca76bed89eddff99d6719f92166ecca9e1036a516497fd555393f6fed7"} Oct 30 22:30:15.509394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:15.509365 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:15.604052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:15.604001 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podStartSLOduration=2.603984562 podStartE2EDuration="2.603984562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:15.60251992 +0000 UTC m=+2595.003076040" watchObservedRunningTime="2026-10-30 22:30:15.603984562 +0000 UTC m=+2595.004540685" Oct 30 22:30:16.493152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.493111 5607 generic.go:356] "Generic (PLEG): container finished" podID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerID="a76ceeca76bed89eddff99d6719f92166ecca9e1036a516497fd555393f6fed7" exitCode=60 Oct 30 22:30:16.493658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.493174 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerDied","Data":"a76ceeca76bed89eddff99d6719f92166ecca9e1036a516497fd555393f6fed7"} Oct 30 22:30:16.493658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.493232 5607 scope.go:122] "RemoveContainer" containerID="7cbd4e642a97a5e973bb887956e2ceb54c7ff4d3e6d20b1c00e30641aeaceddb" Oct 30 22:30:16.493658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.493631 5607 scope.go:122] "RemoveContainer" containerID="a76ceeca76bed89eddff99d6719f92166ecca9e1036a516497fd555393f6fed7" Oct 30 22:30:16.494015 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:16.493968 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8_odh-ai-gateway-infra(d191d4b0-7e8c-4d71-907a-ad533bd96ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" Oct 30 22:30:16.522588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.522552 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:30:16.522850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.522614 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:30:16.734785 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.734751 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8"] Oct 30 22:30:16.863174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.863085 5607 scope.go:122] "RemoveContainer" containerID="8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057" Oct 30 22:30:16.863386 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:16.863365 5607 scope.go:122] "RemoveContainer" containerID="00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3" Oct 30 22:30:17.384472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.384446 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:30:17.528032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.517917 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:30:17.530873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.530837 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:30:17.545920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.545862 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerStarted","Data":"e6d3e59a7d0eac10d3e1dedff760af2ba3d148cffb6e58bf8fe24208f63bda5a"} Oct 30 22:30:17.549906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.549882 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerStarted","Data":"9b3737eb6ddf61e46cef0ddf96589dc8b3759fdd24bb23679041143c2c99832a"} Oct 30 22:30:17.782393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.782243 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:17.863485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.863459 5607 scope.go:122] "RemoveContainer" containerID="9e31e87dc4f576dae2f46eee2c6ab71b156937161e3be4f2cd1c1dcacad18461" Oct 30 22:30:17.886810 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.886788 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575\" (UniqueName: \"kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575\") pod \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " Oct 30 22:30:17.886920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.886847 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca\") pod \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\" (UID: \"d191d4b0-7e8c-4d71-907a-ad533bd96ff0\") " Oct 30 22:30:17.887356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.887334 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca" pod "d191d4b0-7e8c-4d71-907a-ad533bd96ff0" (UID: "d191d4b0-7e8c-4d71-907a-ad533bd96ff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.889202 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.889177 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575" pod "d191d4b0-7e8c-4d71-907a-ad533bd96ff0" (UID: "d191d4b0-7e8c-4d71-907a-ad533bd96ff0"). InnerVolumeSpecName "kube-api-access-vs575". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.992978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.992852 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs575\" (UniqueName: \"kubernetes.io/projected/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-kube-api-access-vs575\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.992978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:17.992887 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191d4b0-7e8c-4d71-907a-ad533bd96ff0-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.525979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.513531 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:30:18.525979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.513610 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:30:18.569579 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.569521 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" event={"ID":"6a37637c-f2e6-477a-904f-95c217afa0d7","Type":"ContainerStarted","Data":"f6d5a11b9ff1aafc1a8e77388443a6b024ecad80e360942ce88e7c714096d94f"} Oct 30 22:30:18.572089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.572058 5607 generic.go:356] "Generic (PLEG): container finished" podID="c8b8223d-c52c-4947-b368-0b85c9438806" containerID="e6d3e59a7d0eac10d3e1dedff760af2ba3d148cffb6e58bf8fe24208f63bda5a" exitCode=60 Oct 30 22:30:18.572241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.572162 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerDied","Data":"e6d3e59a7d0eac10d3e1dedff760af2ba3d148cffb6e58bf8fe24208f63bda5a"} Oct 30 22:30:18.572241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.572211 5607 scope.go:122] "RemoveContainer" containerID="00eb007aff1b22b7d0a0ceb9b1a5c8bace55806377bc94d3bb6aa321b95654a3" Oct 30 22:30:18.572646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.572627 5607 scope.go:122] "RemoveContainer" containerID="e6d3e59a7d0eac10d3e1dedff760af2ba3d148cffb6e58bf8fe24208f63bda5a" Oct 30 22:30:18.576360 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:18.576319 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch_odh-ai-gateway-infra(c8b8223d-c52c-4947-b368-0b85c9438806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" Oct 30 22:30:18.576738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.576711 5607 generic.go:356] "Generic (PLEG): container finished" podID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerID="9b3737eb6ddf61e46cef0ddf96589dc8b3759fdd24bb23679041143c2c99832a" exitCode=60 Oct 30 22:30:18.576841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.576810 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerDied","Data":"9b3737eb6ddf61e46cef0ddf96589dc8b3759fdd24bb23679041143c2c99832a"} Oct 30 22:30:18.577828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.577806 5607 scope.go:122] "RemoveContainer" containerID="9b3737eb6ddf61e46cef0ddf96589dc8b3759fdd24bb23679041143c2c99832a" Oct 30 22:30:18.578295 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:18.578271 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g_odh-ai-gateway-infra(91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" Oct 30 22:30:18.579830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.579808 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" event={"ID":"d191d4b0-7e8c-4d71-907a-ad533bd96ff0","Type":"ContainerDied","Data":"cd6f550bbb5c651b588685070b2e7792bcb9d304b89b4777287ed35271d4c8b0"} Oct 30 22:30:18.580065 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.580048 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8" Oct 30 22:30:18.582045 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.582020 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst"] Oct 30 22:30:18.582967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.582928 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.582967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.582965 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.583130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.582980 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.583130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.582987 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.583248 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.583184 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.583248 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.583205 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.583248 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.583214 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:18.587329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.587309 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.597112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.597092 5607 scope.go:122] "RemoveContainer" containerID="8adefbe630965040ca5d59199e643a9ca35e10e3cc08fa031412739eda12c057" Oct 30 22:30:18.636584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.636566 5607 scope.go:122] "RemoveContainer" containerID="a76ceeca76bed89eddff99d6719f92166ecca9e1036a516497fd555393f6fed7" Oct 30 22:30:18.643181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.643122 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f39eb7-dockercfg-wgrwj\"" Oct 30 22:30:18.643181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.643165 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f39eb7\"" Oct 30 22:30:18.657270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.657236 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst"] Oct 30 22:30:18.721532 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.721430 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch"] Oct 30 22:30:18.726336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.726287 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" podStartSLOduration=18.726274838 podStartE2EDuration="18.726274838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:18.723619622 +0000 UTC m=+2598.124175743" watchObservedRunningTime="2026-10-30 22:30:18.726274838 +0000 UTC m=+2598.126830950" Oct 30 22:30:18.757883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.757853 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.758225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.758195 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frczb\" (UniqueName: \"kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb\") pod \"payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.839343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.839268 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8"] Oct 30 22:30:18.847609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.847580 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-c301cf-29889990-rd6k8"] Oct 30 22:30:18.859666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.859635 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.859775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.859700 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frczb\" (UniqueName: \"kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb\") pod \"payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.860326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.860306 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.870619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.870598 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" path="/var/lib/kubelet/pods/d191d4b0-7e8c-4d71-907a-ad533bd96ff0/volumes" Oct 30 22:30:18.924442 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.924414 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frczb\" (UniqueName: \"kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb\") pod \"payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.931447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.931424 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:18.936662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:18.936639 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g"] Oct 30 22:30:19.338380 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.338354 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst"] Oct 30 22:30:19.520979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.513332 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:30:19.520979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.513388 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:30:19.591619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.591530 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" event={"ID":"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db","Type":"ContainerStarted","Data":"342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861"} Oct 30 22:30:19.591619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.591592 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" event={"ID":"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db","Type":"ContainerStarted","Data":"cbac28d8ec491dc9a5ff7f41b9f80dc65fb63c2bd503f87999c3a72167f4a324"} Oct 30 22:30:19.592204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.591758 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:19.593747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.593722 5607 generic.go:356] "Generic (PLEG): container finished" podID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerID="f6d5a11b9ff1aafc1a8e77388443a6b024ecad80e360942ce88e7c714096d94f" exitCode=60 Oct 30 22:30:19.593893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.593831 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" event={"ID":"6a37637c-f2e6-477a-904f-95c217afa0d7","Type":"ContainerDied","Data":"f6d5a11b9ff1aafc1a8e77388443a6b024ecad80e360942ce88e7c714096d94f"} Oct 30 22:30:19.593893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.593864 5607 scope.go:122] "RemoveContainer" containerID="9e31e87dc4f576dae2f46eee2c6ab71b156937161e3be4f2cd1c1dcacad18461" Oct 30 22:30:19.594187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.594167 5607 scope.go:122] "RemoveContainer" containerID="f6d5a11b9ff1aafc1a8e77388443a6b024ecad80e360942ce88e7c714096d94f" Oct 30 22:30:19.594430 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:19.594409 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6_odh-ai-gateway-infra(6a37637c-f2e6-477a-904f-95c217afa0d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" Oct 30 22:30:19.735002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.732594 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" podStartSLOduration=1.7325774699999998 podStartE2EDuration="1.73257747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:19.730016865 +0000 UTC m=+2599.130572986" watchObservedRunningTime="2026-10-30 22:30:19.73257747 +0000 UTC m=+2599.133133644" Oct 30 22:30:19.737560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.737528 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6"] Oct 30 22:30:19.862212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.862181 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:19.887229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.886094 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84\" (UniqueName: \"kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84\") pod \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " Oct 30 22:30:19.887229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.886159 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca\") pod \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\" (UID: \"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f\") " Oct 30 22:30:19.887837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.887809 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca" pod "91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" (UID: "91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.888488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.888472 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84" pod "91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" (UID: "91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f"). InnerVolumeSpecName "kube-api-access-tmk84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.928517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.928486 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:19.988197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.987436 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca\") pod \"c8b8223d-c52c-4947-b368-0b85c9438806\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " Oct 30 22:30:19.988197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.987564 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn\" (UniqueName: \"kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn\") pod \"c8b8223d-c52c-4947-b368-0b85c9438806\" (UID: \"c8b8223d-c52c-4947-b368-0b85c9438806\") " Oct 30 22:30:19.988197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.987805 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca" pod "c8b8223d-c52c-4947-b368-0b85c9438806" (UID: "c8b8223d-c52c-4947-b368-0b85c9438806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.988488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.988207 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b8223d-c52c-4947-b368-0b85c9438806-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.988488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.988232 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmk84\" (UniqueName: \"kubernetes.io/projected/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-kube-api-access-tmk84\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.988488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.988248 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.990605 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:19.990581 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn" pod "c8b8223d-c52c-4947-b368-0b85c9438806" (UID: "c8b8223d-c52c-4947-b368-0b85c9438806"). InnerVolumeSpecName "kube-api-access-n9ckn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.089690 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.089588 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ckn\" (UniqueName: \"kubernetes.io/projected/c8b8223d-c52c-4947-b368-0b85c9438806-kube-api-access-n9ckn\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.521002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.520370 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:30:20.521002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.520429 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:30:20.635629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.635598 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" event={"ID":"c8b8223d-c52c-4947-b368-0b85c9438806","Type":"ContainerDied","Data":"597684d2ce078d9618e3f7b801fbd094df6197b85b948d8901331094a12f22a9"} Oct 30 22:30:20.636064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.635661 5607 scope.go:122] "RemoveContainer" containerID="e6d3e59a7d0eac10d3e1dedff760af2ba3d148cffb6e58bf8fe24208f63bda5a" Oct 30 22:30:20.636064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.635755 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch" Oct 30 22:30:20.638013 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.637982 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" event={"ID":"91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f","Type":"ContainerDied","Data":"c6def12f14fb7ce2c0f18a182f9cd202c9ef5f07efc422822608e9bdcc1c5c6a"} Oct 30 22:30:20.638079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.638001 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g" Oct 30 22:30:20.660654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.660627 5607 scope.go:122] "RemoveContainer" containerID="9b3737eb6ddf61e46cef0ddf96589dc8b3759fdd24bb23679041143c2c99832a" Oct 30 22:30:20.832083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.832053 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g"] Oct 30 22:30:20.836730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.836617 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889990-bcd8g"] Oct 30 22:30:20.862906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.862826 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:20.876160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.876136 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" path="/var/lib/kubelet/pods/91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f/volumes" Oct 30 22:30:20.929310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.929182 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg\" (UniqueName: \"kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg\") pod \"6a37637c-f2e6-477a-904f-95c217afa0d7\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " Oct 30 22:30:20.929462 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.929386 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca\") pod \"6a37637c-f2e6-477a-904f-95c217afa0d7\" (UID: \"6a37637c-f2e6-477a-904f-95c217afa0d7\") " Oct 30 22:30:20.931873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.930202 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca" pod "6a37637c-f2e6-477a-904f-95c217afa0d7" (UID: "6a37637c-f2e6-477a-904f-95c217afa0d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.933365 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.933099 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg" pod "6a37637c-f2e6-477a-904f-95c217afa0d7" (UID: "6a37637c-f2e6-477a-904f-95c217afa0d7"). InnerVolumeSpecName "kube-api-access-9q9tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.941664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.941643 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch"] Oct 30 22:30:20.948973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:20.948936 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1875b4-29889990-vl5ch"] Oct 30 22:30:21.031336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.031303 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9tg\" (UniqueName: \"kubernetes.io/projected/6a37637c-f2e6-477a-904f-95c217afa0d7-kube-api-access-9q9tg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.031336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.031331 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a37637c-f2e6-477a-904f-95c217afa0d7-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.509922 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.509885 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 22:30:21.510117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.509939 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 22:30:21.651465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.651437 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" Oct 30 22:30:21.651904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.651433 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6" event={"ID":"6a37637c-f2e6-477a-904f-95c217afa0d7","Type":"ContainerDied","Data":"eb73854502c4853adba14bb36a80c2cb8bc913f808b06fdbb5b49b1d0d9d98ce"} Oct 30 22:30:21.651904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.651575 5607 scope.go:122] "RemoveContainer" containerID="f6d5a11b9ff1aafc1a8e77388443a6b024ecad80e360942ce88e7c714096d94f" Oct 30 22:30:21.831991 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.831873 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6"] Oct 30 22:30:21.842835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:21.842809 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-28f124-29889990-dffd6"] Oct 30 22:30:22.605486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:22.605458 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:22.605929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:22.605912 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:22.606788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:22.606770 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:30:22.863805 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:22.863725 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:30:22.870940 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:22.870917 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" path="/var/lib/kubelet/pods/6a37637c-f2e6-477a-904f-95c217afa0d7/volumes" Oct 30 22:30:22.871364 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:22.871350 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" path="/var/lib/kubelet/pods/c8b8223d-c52c-4947-b368-0b85c9438806/volumes" Oct 30 22:30:29.027623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.027585 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b"] Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028321 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028338 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028353 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028357 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028369 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028376 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028384 5607 cpu_manager.go:414] "containerMap: removing container" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028389 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028399 5607 cpu_manager.go:414] "containerMap: removing container" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028403 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028416 5607 cpu_manager.go:414] "containerMap: removing container" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028423 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028430 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028434 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028441 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028445 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028483 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028488 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d191d4b0-7e8c-4d71-907a-ad533bd96ff0" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028570 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028582 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028590 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028595 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028601 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:29.030163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.028607 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:29.032278 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.032263 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.132469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.132438 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b"] Oct 30 22:30:29.136604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.136571 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.136790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.136658 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.136790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.136732 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.136790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.136767 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.137000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.136833 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9s5z\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.137000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.136863 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.137132 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.137013 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.137132 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.137075 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.137262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.137137 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238155 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238221 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238368 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238272 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238394 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238440 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238465 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238532 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9s5z\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238566 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238587 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238710 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.238802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.238724 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.239212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.239188 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.239270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.239211 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.239316 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.239300 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.240722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.240699 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.240913 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.240892 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.320438 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.320369 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.320982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.320927 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9s5z\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.345962 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.345937 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:29.812600 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:29.812570 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69672d21_b1d9_49ed_bf6b_a92596d2dd04.slice/crio-7038e0298513674112d7e403712852433387e5664e0a17f83f085d632ce0f38b WatchSource:0}: Error finding container 7038e0298513674112d7e403712852433387e5664e0a17f83f085d632ce0f38b: Status 404 returned error can't find the container with id 7038e0298513674112d7e403712852433387e5664e0a17f83f085d632ce0f38b Oct 30 22:30:29.820100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.816850 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b"] Oct 30 22:30:29.822281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.822249 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:29.822369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.822317 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:29.822369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:29.822346 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:30.644079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:30.644047 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:30:30.737572 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:30.737533 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" event={"ID":"69672d21-b1d9-49ed-bf6b-a92596d2dd04","Type":"ContainerStarted","Data":"c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53"} Oct 30 22:30:30.737782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:30.737592 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" event={"ID":"69672d21-b1d9-49ed-bf6b-a92596d2dd04","Type":"ContainerStarted","Data":"7038e0298513674112d7e403712852433387e5664e0a17f83f085d632ce0f38b"} Oct 30 22:30:30.910035 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:30.900971 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podStartSLOduration=2.900929696 podStartE2EDuration="2.900929696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:30.894679039 +0000 UTC m=+2610.295235160" watchObservedRunningTime="2026-10-30 22:30:30.900929696 +0000 UTC m=+2610.301485819" Oct 30 22:30:31.346324 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:31.346281 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:31.348119 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:31.348091 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:30:31.348259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:31.348154 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:30:32.347028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:32.346940 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:30:32.347434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:32.347051 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:30:33.346599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:33.346557 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:30:33.346782 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:33.346607 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:30:34.347118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:34.347084 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:30:34.347607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:34.347153 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:30:35.347114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:35.347081 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:30:35.347310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:35.347136 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:30:35.821992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:35.820057 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:30:35.821992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:35.820120 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:30:36.346746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:36.346706 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 22:30:36.346927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:36.346765 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 22:30:37.764889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.764806 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:37.765414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.765272 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:37.766022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.766003 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:30:37.870431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.868534 5607 generic.go:356] "Generic (PLEG): container finished" podID="2064e2f8-2156-47bc-9601-82874abb8df7" containerID="7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37" exitCode=0 Oct 30 22:30:37.870431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.869539 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" event={"ID":"2064e2f8-2156-47bc-9601-82874abb8df7","Type":"ContainerDied","Data":"7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37"} Oct 30 22:30:37.870431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.869912 5607 scope.go:122] "RemoveContainer" containerID="7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37" Oct 30 22:30:37.873717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.872864 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:37.873717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.872988 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:37.873717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:37.873038 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:30:38.968047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:38.967128 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" event={"ID":"2064e2f8-2156-47bc-9601-82874abb8df7","Type":"ContainerStarted","Data":"5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7"} Oct 30 22:30:39.550979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:39.550920 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:30:39.551216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:39.551074 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:30:39.551216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:39.551119 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:30:40.547770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:40.547689 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:30:40.547770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:40.547747 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:30:41.548209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:41.548175 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:30:41.548613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:41.548234 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:30:42.548468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:42.548433 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:30:42.548862 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:42.548490 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:30:42.921416 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:42.921310 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4"] Oct 30 22:30:43.028808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:43.028773 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh"] Oct 30 22:30:43.548150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:43.548104 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:30:43.548393 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:43.548175 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:30:44.549707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:44.549668 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 22:30:44.550267 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:44.549734 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 22:30:45.552236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:45.552206 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:30:45.552744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:45.552543 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:30:45.553153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:45.553132 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:30:47.637517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.637479 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh"] Oct 30 22:30:47.655994 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.655943 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:47.657147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.657125 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:47.657619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.657601 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a37637c-f2e6-477a-904f-95c217afa0d7" containerName="cleanup" Oct 30 22:30:47.659026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.659010 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="91dafc1a-fe49-4b34-95f0-c8d5f39a6d6f" containerName="cleanup" Oct 30 22:30:47.659183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.659171 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b8223d-c52c-4947-b368-0b85c9438806" containerName="cleanup" Oct 30 22:30:47.671491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.671467 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.700785 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.700757 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh"] Oct 30 22:30:47.722115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.722085 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vgz\" (UniqueName: \"kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz\") pod \"payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.722485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.722228 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.747177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.747145 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fc6466\"" Oct 30 22:30:47.747398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.747376 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fc6466-dockercfg-lnth8\"" Oct 30 22:30:47.823729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.823696 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vgz\" (UniqueName: \"kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz\") pod \"payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.823940 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.823818 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.824578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.824560 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.886768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.886740 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vgz\" (UniqueName: \"kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz\") pod \"payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:47.922181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.922103 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" podUID="c51ef35f-3e75-4950-8644-3e388e6ebac5" containerName="payload-processing" containerID="cri-o://d37de93cc2989b5e26606a5e1dc65dcb4b8e8b3dbeff9c812a326b170e0982aa" gracePeriod=25 Oct 30 22:30:47.987541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:47.987517 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:48.029722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.029674 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" podUID="be3c0fd0-9df5-4109-bc54-1951a0427430" containerName="payload-pre-processing" containerID="cri-o://11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513" gracePeriod=25 Oct 30 22:30:48.068253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.068225 5607 generic.go:356] "Generic (PLEG): container finished" podID="c51ef35f-3e75-4950-8644-3e388e6ebac5" containerID="d37de93cc2989b5e26606a5e1dc65dcb4b8e8b3dbeff9c812a326b170e0982aa" exitCode=0 Oct 30 22:30:48.068427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.068276 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" event={"ID":"c51ef35f-3e75-4950-8644-3e388e6ebac5","Type":"ContainerDied","Data":"d37de93cc2989b5e26606a5e1dc65dcb4b8e8b3dbeff9c812a326b170e0982aa"} Oct 30 22:30:48.217375 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.217342 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:30:48.311051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.311019 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:30:48.330706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.330670 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume\") pod \"c51ef35f-3e75-4950-8644-3e388e6ebac5\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " Oct 30 22:30:48.330856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.330787 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w\" (UniqueName: \"kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w\") pod \"c51ef35f-3e75-4950-8644-3e388e6ebac5\" (UID: \"c51ef35f-3e75-4950-8644-3e388e6ebac5\") " Oct 30 22:30:48.331037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.331016 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume" pod "c51ef35f-3e75-4950-8644-3e388e6ebac5" (UID: "c51ef35f-3e75-4950-8644-3e388e6ebac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:48.331342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.331326 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51ef35f-3e75-4950-8644-3e388e6ebac5-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:48.333187 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.333162 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w" pod "c51ef35f-3e75-4950-8644-3e388e6ebac5" (UID: "c51ef35f-3e75-4950-8644-3e388e6ebac5"). InnerVolumeSpecName "kube-api-access-4rn6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:48.369726 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:48.369701 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b73df7_7c61_4a96_8813_cb5b3548b31c.slice/crio-3954c3cb7e6571905ffad5dc1f738f2c51a9a026c98a7dba9a8e77349cbbc2e4 WatchSource:0}: Error finding container 3954c3cb7e6571905ffad5dc1f738f2c51a9a026c98a7dba9a8e77349cbbc2e4: Status 404 returned error can't find the container with id 3954c3cb7e6571905ffad5dc1f738f2c51a9a026c98a7dba9a8e77349cbbc2e4 Oct 30 22:30:48.392283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.392252 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh"] Oct 30 22:30:48.440151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.436942 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume\") pod \"be3c0fd0-9df5-4109-bc54-1951a0427430\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " Oct 30 22:30:48.440151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.437167 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv\" (UniqueName: \"kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv\") pod \"be3c0fd0-9df5-4109-bc54-1951a0427430\" (UID: \"be3c0fd0-9df5-4109-bc54-1951a0427430\") " Oct 30 22:30:48.440151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.437744 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rn6w\" (UniqueName: \"kubernetes.io/projected/c51ef35f-3e75-4950-8644-3e388e6ebac5-kube-api-access-4rn6w\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:48.440151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.438357 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume" pod "be3c0fd0-9df5-4109-bc54-1951a0427430" (UID: "be3c0fd0-9df5-4109-bc54-1951a0427430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:48.441457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.441417 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv" pod "be3c0fd0-9df5-4109-bc54-1951a0427430" (UID: "be3c0fd0-9df5-4109-bc54-1951a0427430"). InnerVolumeSpecName "kube-api-access-tc5nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:48.539383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.539350 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c0fd0-9df5-4109-bc54-1951a0427430-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:48.539383 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:48.539376 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5nv\" (UniqueName: \"kubernetes.io/projected/be3c0fd0-9df5-4109-bc54-1951a0427430-kube-api-access-tc5nv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:49.076297 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.076259 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" event={"ID":"30b73df7-7c61-4a96-8813-cb5b3548b31c","Type":"ContainerStarted","Data":"144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168"} Oct 30 22:30:49.076779 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.076309 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" event={"ID":"30b73df7-7c61-4a96-8813-cb5b3548b31c","Type":"ContainerStarted","Data":"3954c3cb7e6571905ffad5dc1f738f2c51a9a026c98a7dba9a8e77349cbbc2e4"} Oct 30 22:30:49.076779 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.076339 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:30:49.077849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.077817 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" Oct 30 22:30:49.078014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.077859 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4" event={"ID":"c51ef35f-3e75-4950-8644-3e388e6ebac5","Type":"ContainerDied","Data":"fe5e131f534e28efa3a17a692d1b7d35cbb992f9d7c6406b40d6d180081be15e"} Oct 30 22:30:49.078014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.077916 5607 scope.go:122] "RemoveContainer" containerID="d37de93cc2989b5e26606a5e1dc65dcb4b8e8b3dbeff9c812a326b170e0982aa" Oct 30 22:30:49.079784 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.079759 5607 generic.go:356] "Generic (PLEG): container finished" podID="be3c0fd0-9df5-4109-bc54-1951a0427430" containerID="11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513" exitCode=0 Oct 30 22:30:49.079892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.079789 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" Oct 30 22:30:49.079892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.079822 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" event={"ID":"be3c0fd0-9df5-4109-bc54-1951a0427430","Type":"ContainerDied","Data":"11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513"} Oct 30 22:30:49.079892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.079864 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh" event={"ID":"be3c0fd0-9df5-4109-bc54-1951a0427430","Type":"ContainerDied","Data":"f5c49525c7c82e2bdab281618d45d61e106c29442367922937b6c2b629695b19"} Oct 30 22:30:49.092121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.092099 5607 scope.go:122] "RemoveContainer" containerID="11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513" Oct 30 22:30:49.101409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.101388 5607 scope.go:122] "RemoveContainer" containerID="11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513" Oct 30 22:30:49.101693 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:49.101669 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513\": container with ID starting with 11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513 not found: ID does not exist" containerID="11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513" Oct 30 22:30:49.101754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.101703 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513"} err="failed to get container status \"11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513\": rpc error: code = NotFound desc = could not find container \"11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513\": container with ID starting with 11e474df8fca205ccff5cf2b6ca2adf38427d59ea02a55d9ce41b415eb343513 not found: ID does not exist" Oct 30 22:30:49.312685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.312631 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" podStartSLOduration=2.312617629 podStartE2EDuration="2.312617629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:49.200050818 +0000 UTC m=+2628.600606940" watchObservedRunningTime="2026-10-30 22:30:49.312617629 +0000 UTC m=+2628.713173744" Oct 30 22:30:49.315725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.315695 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh"] Oct 30 22:30:49.320398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.320373 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-459aa8-58d457ff5f-7vbfh"] Oct 30 22:30:49.431043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.431003 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4"] Oct 30 22:30:49.435517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:49.435491 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-459aa8-5dff44787f-9jnb4"] Oct 30 22:30:50.872032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:50.871998 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3c0fd0-9df5-4109-bc54-1951a0427430" path="/var/lib/kubelet/pods/be3c0fd0-9df5-4109-bc54-1951a0427430/volumes" Oct 30 22:30:50.872412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:50.872381 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51ef35f-3e75-4950-8644-3e388e6ebac5" path="/var/lib/kubelet/pods/c51ef35f-3e75-4950-8644-3e388e6ebac5/volumes" Oct 30 22:30:52.864795 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:52.864757 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:30:53.719895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.719861 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp"] Oct 30 22:30:53.720969 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.720931 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c51ef35f-3e75-4950-8644-3e388e6ebac5" containerName="payload-processing" Oct 30 22:30:53.721024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.720972 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51ef35f-3e75-4950-8644-3e388e6ebac5" containerName="payload-processing" Oct 30 22:30:53.721024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.721007 5607 cpu_manager.go:414] "containerMap: removing container" podUID="be3c0fd0-9df5-4109-bc54-1951a0427430" containerName="payload-pre-processing" Oct 30 22:30:53.721024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.721016 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3c0fd0-9df5-4109-bc54-1951a0427430" containerName="payload-pre-processing" Oct 30 22:30:53.721178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.721165 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51ef35f-3e75-4950-8644-3e388e6ebac5" containerName="payload-processing" Oct 30 22:30:53.721218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.721188 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3c0fd0-9df5-4109-bc54-1951a0427430" containerName="payload-pre-processing" Oct 30 22:30:53.725101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.725080 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.805901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.805851 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.806135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.805916 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2kxq\" (UniqueName: \"kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.806135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.806030 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.806135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.806078 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.847820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.846786 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-9717a8-serving-cert\"" Oct 30 22:30:53.847820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.847506 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp"] Oct 30 22:30:53.908186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.908153 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.908778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.908750 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2kxq\" (UniqueName: \"kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.909021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.908999 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.909206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.909188 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.909403 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:53.908351 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-serving-cert: secret "maas-api-e2e-shared-a-w2-9717a8-serving-cert" not found Oct 30 22:30:53.909488 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:30:53.909474 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls podName:09f8a61b-d445-4b2d-b66c-e37b18c93114 nodeName:}" failed. No retries permitted until 2026-10-30 22:30:54.409450842 +0000 UTC m=+2633.810006965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls") pod "maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" (UID: "09f8a61b-d445-4b2d-b66c-e37b18c93114") : secret "maas-api-e2e-shared-a-w2-9717a8-serving-cert" not found Oct 30 22:30:53.910617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.910592 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.937924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.931711 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:53.993054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:53.990842 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2kxq\" (UniqueName: \"kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:54.426885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:54.426797 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:54.446585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:54.446263 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:54.640429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:54.640400 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:55.013441 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:30:55.013388 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f8a61b_d445_4b2d_b66c_e37b18c93114.slice/crio-96cf1dba484ab1ed99467da03f20cf3a389bb003b8295e1a41a7b79c25400926 WatchSource:0}: Error finding container 96cf1dba484ab1ed99467da03f20cf3a389bb003b8295e1a41a7b79c25400926: Status 404 returned error can't find the container with id 96cf1dba484ab1ed99467da03f20cf3a389bb003b8295e1a41a7b79c25400926 Oct 30 22:30:55.017117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:55.016872 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp"] Oct 30 22:30:55.131911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:55.131882 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" event={"ID":"09f8a61b-d445-4b2d-b66c-e37b18c93114","Type":"ContainerStarted","Data":"96cf1dba484ab1ed99467da03f20cf3a389bb003b8295e1a41a7b79c25400926"} Oct 30 22:30:56.141028 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:56.140992 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" event={"ID":"09f8a61b-d445-4b2d-b66c-e37b18c93114","Type":"ContainerStarted","Data":"cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1"} Oct 30 22:30:56.141484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:56.141059 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:30:56.271787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:30:56.271705 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" podStartSLOduration=2.319960769 podStartE2EDuration="3.271691215s" totalImagesPullingTime="951.730446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:56.269402887 +0000 UTC m=+2635.669959008" watchObservedRunningTime="2026-10-30 22:30:56.271691215 +0000 UTC m=+2635.672247337" Oct 30 22:31:00.093095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:00.093069 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:31:02.153420 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:02.153394 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:31:02.687833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:02.687802 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s"] Oct 30 22:31:02.688054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:02.688013 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" podUID="0eb0bc20-8039-48d8-9ab3-6649188a7bf8" containerName="maas-api" containerID="cri-o://c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5" gracePeriod=30 Oct 30 22:31:02.968226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:02.968197 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:31:03.132811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.132784 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k\" (UniqueName: \"kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k\") pod \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " Oct 30 22:31:03.133007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.132829 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca\") pod \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " Oct 30 22:31:03.133007 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.132897 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\") pod \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " Oct 30 22:31:03.133102 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.133078 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs\") pod \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\" (UID: \"0eb0bc20-8039-48d8-9ab3-6649188a7bf8\") " Oct 30 22:31:03.133306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.133279 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca" pod "0eb0bc20-8039-48d8-9ab3-6649188a7bf8" (UID: "0eb0bc20-8039-48d8-9ab3-6649188a7bf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:03.133564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.133546 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:03.135233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.135204 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls" pod "0eb0bc20-8039-48d8-9ab3-6649188a7bf8" (UID: "0eb0bc20-8039-48d8-9ab3-6649188a7bf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:03.135333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.135277 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k" pod "0eb0bc20-8039-48d8-9ab3-6649188a7bf8" (UID: "0eb0bc20-8039-48d8-9ab3-6649188a7bf8"). InnerVolumeSpecName "kube-api-access-2fx8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:03.135644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.135627 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs" pod "0eb0bc20-8039-48d8-9ab3-6649188a7bf8" (UID: "0eb0bc20-8039-48d8-9ab3-6649188a7bf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:03.185546 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.185521 5607 generic.go:356] "Generic (PLEG): container finished" podID="0eb0bc20-8039-48d8-9ab3-6649188a7bf8" containerID="c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5" exitCode=0 Oct 30 22:31:03.185896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.185567 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" event={"ID":"0eb0bc20-8039-48d8-9ab3-6649188a7bf8","Type":"ContainerDied","Data":"c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5"} Oct 30 22:31:03.185896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.185572 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" Oct 30 22:31:03.185896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.185611 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s" event={"ID":"0eb0bc20-8039-48d8-9ab3-6649188a7bf8","Type":"ContainerDied","Data":"2a889b56b1b3b64454de18fa05c5c484f8909431aa3fabe8b57fe53c1ebbf2eb"} Oct 30 22:31:03.185896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.185638 5607 scope.go:122] "RemoveContainer" containerID="c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5" Oct 30 22:31:03.196018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.195995 5607 scope.go:122] "RemoveContainer" containerID="c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5" Oct 30 22:31:03.196283 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:03.196265 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5\": container with ID starting with c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5 not found: ID does not exist" containerID="c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5" Oct 30 22:31:03.196350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.196290 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5"} err="failed to get container status \"c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5\": rpc error: code = NotFound desc = could not find container \"c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5\": container with ID starting with c4457f83a29e9458165af06ec5f787607cfa40274bc746dff267933526d411b5 not found: ID does not exist" Oct 30 22:31:03.238977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.238204 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:03.238977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.238241 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fx8k\" (UniqueName: \"kubernetes.io/projected/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-kube-api-access-2fx8k\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:03.238977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.238260 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb0bc20-8039-48d8-9ab3-6649188a7bf8-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:03.357997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.357965 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s"] Oct 30 22:31:03.361533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:03.361509 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-459aa8-665dfc6b5b-4456s"] Oct 30 22:31:04.186897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.186860 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh"] Oct 30 22:31:04.187537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.187516 5607 cpu_manager.go:414] "containerMap: removing container" podUID="0eb0bc20-8039-48d8-9ab3-6649188a7bf8" containerName="maas-api" Oct 30 22:31:04.187537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.187537 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb0bc20-8039-48d8-9ab3-6649188a7bf8" containerName="maas-api" Oct 30 22:31:04.187718 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.187706 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb0bc20-8039-48d8-9ab3-6649188a7bf8" containerName="maas-api" Oct 30 22:31:04.191116 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.191099 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.244725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.244704 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f39eb7-serving-cert\"" Oct 30 22:31:04.298736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.298712 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh"] Oct 30 22:31:04.353921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.353878 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtl6s\" (UniqueName: \"kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.354125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.354106 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.354189 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.354143 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.354264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.354236 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.456307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.456209 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtl6s\" (UniqueName: \"kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.456568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.456418 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.456568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.456455 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.456568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.456507 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.456918 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:04.456896 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-serving-cert: secret "maas-api-e2e-shared-a-w4-f39eb7-serving-cert" not found Oct 30 22:31:04.457081 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:04.457065 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls podName:50e54bdf-abbb-4a76-ae6e-c98d237359d1 nodeName:}" failed. No retries permitted until 2026-10-30 22:31:04.957042008 +0000 UTC m=+2644.357598108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls") pod "maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" (UID: "50e54bdf-abbb-4a76-ae6e-c98d237359d1") : secret "maas-api-e2e-shared-a-w4-f39eb7-serving-cert" not found Oct 30 22:31:04.458736 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.457541 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.461437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.461414 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.521757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.521733 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtl6s\" (UniqueName: \"kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.870398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.870324 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb0bc20-8039-48d8-9ab3-6649188a7bf8" path="/var/lib/kubelet/pods/0eb0bc20-8039-48d8-9ab3-6649188a7bf8/volumes" Oct 30 22:31:04.962752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.962723 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:04.965381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:04.965356 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:05.103997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:05.103965 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:05.474326 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:31:05.474291 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e54bdf_abbb_4a76_ae6e_c98d237359d1.slice/crio-72ae8a80822eaf7181bdf1e6f5992790982d0ac0cb6b5ae13cbc35413601cf7e WatchSource:0}: Error finding container 72ae8a80822eaf7181bdf1e6f5992790982d0ac0cb6b5ae13cbc35413601cf7e: Status 404 returned error can't find the container with id 72ae8a80822eaf7181bdf1e6f5992790982d0ac0cb6b5ae13cbc35413601cf7e Oct 30 22:31:05.477707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:05.477687 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh"] Oct 30 22:31:06.209781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:06.209745 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" event={"ID":"50e54bdf-abbb-4a76-ae6e-c98d237359d1","Type":"ContainerStarted","Data":"72ae8a80822eaf7181bdf1e6f5992790982d0ac0cb6b5ae13cbc35413601cf7e"} Oct 30 22:31:07.218632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:07.218594 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" event={"ID":"50e54bdf-abbb-4a76-ae6e-c98d237359d1","Type":"ContainerStarted","Data":"0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9"} Oct 30 22:31:07.219120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:07.218669 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:07.340685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:07.340625 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" podStartSLOduration=2.451786906 podStartE2EDuration="3.340611609s" totalImagesPullingTime="888.824703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:07.338201082 +0000 UTC m=+2646.738757200" watchObservedRunningTime="2026-10-30 22:31:07.340611609 +0000 UTC m=+2646.741167730" Oct 30 22:31:13.232054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:13.232025 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:31:22.866885 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:22.866845 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:31:24.291440 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:24.291410 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx"] Oct 30 22:31:24.407307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:24.407278 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp"] Oct 30 22:31:29.291826 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.291788 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" podUID="a7dcd691-fe9b-4b17-addc-504be4367090" containerName="payload-processing" containerID="cri-o://8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552" gracePeriod=25 Oct 30 22:31:29.408505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.408464 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" podUID="bc7f4ee1-1516-4cf6-a581-72db7e851a69" containerName="payload-pre-processing" containerID="cri-o://cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287" gracePeriod=25 Oct 30 22:31:29.571324 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.571296 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:31:29.674350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.674323 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume\") pod \"a7dcd691-fe9b-4b17-addc-504be4367090\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " Oct 30 22:31:29.674507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.674486 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678\" (UniqueName: \"kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678\") pod \"a7dcd691-fe9b-4b17-addc-504be4367090\" (UID: \"a7dcd691-fe9b-4b17-addc-504be4367090\") " Oct 30 22:31:29.674722 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.674700 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume" pod "a7dcd691-fe9b-4b17-addc-504be4367090" (UID: "a7dcd691-fe9b-4b17-addc-504be4367090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:29.675110 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.675088 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7dcd691-fe9b-4b17-addc-504be4367090-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.676621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.676597 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678" pod "a7dcd691-fe9b-4b17-addc-504be4367090" (UID: "a7dcd691-fe9b-4b17-addc-504be4367090"). InnerVolumeSpecName "kube-api-access-4p678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:29.691153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.691133 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:31:29.775833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.775810 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume\") pod \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " Oct 30 22:31:29.776002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.775892 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz\" (UniqueName: \"kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz\") pod \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\" (UID: \"bc7f4ee1-1516-4cf6-a581-72db7e851a69\") " Oct 30 22:31:29.776255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.776239 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p678\" (UniqueName: \"kubernetes.io/projected/a7dcd691-fe9b-4b17-addc-504be4367090-kube-api-access-4p678\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.776295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.776252 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume" pod "bc7f4ee1-1516-4cf6-a581-72db7e851a69" (UID: "bc7f4ee1-1516-4cf6-a581-72db7e851a69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:29.778133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.778115 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz" pod "bc7f4ee1-1516-4cf6-a581-72db7e851a69" (UID: "bc7f4ee1-1516-4cf6-a581-72db7e851a69"). InnerVolumeSpecName "kube-api-access-c9fqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:29.877916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.877848 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc7f4ee1-1516-4cf6-a581-72db7e851a69-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.877916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:29.877872 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9fqz\" (UniqueName: \"kubernetes.io/projected/bc7f4ee1-1516-4cf6-a581-72db7e851a69-kube-api-access-c9fqz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:30.369643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.369611 5607 generic.go:356] "Generic (PLEG): container finished" podID="a7dcd691-fe9b-4b17-addc-504be4367090" containerID="8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552" exitCode=0 Oct 30 22:31:30.370076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.369648 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" Oct 30 22:31:30.370076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.369690 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" event={"ID":"a7dcd691-fe9b-4b17-addc-504be4367090","Type":"ContainerDied","Data":"8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552"} Oct 30 22:31:30.370076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.369742 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx" event={"ID":"a7dcd691-fe9b-4b17-addc-504be4367090","Type":"ContainerDied","Data":"d316f0fbdbaf4c48a395578b24db3c47704a218339580632b70003e2d68efb05"} Oct 30 22:31:30.370076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.369768 5607 scope.go:122] "RemoveContainer" containerID="8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552" Oct 30 22:31:30.371541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.371516 5607 generic.go:356] "Generic (PLEG): container finished" podID="bc7f4ee1-1516-4cf6-a581-72db7e851a69" containerID="cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287" exitCode=0 Oct 30 22:31:30.371656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.371602 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" event={"ID":"bc7f4ee1-1516-4cf6-a581-72db7e851a69","Type":"ContainerDied","Data":"cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287"} Oct 30 22:31:30.371656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.371615 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" Oct 30 22:31:30.371656 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.371644 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp" event={"ID":"bc7f4ee1-1516-4cf6-a581-72db7e851a69","Type":"ContainerDied","Data":"7051ecf4ea57550588c7c2c5d544df0f19cf24be620c297906f4f433c2aef037"} Oct 30 22:31:30.380454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.380429 5607 scope.go:122] "RemoveContainer" containerID="8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552" Oct 30 22:31:30.382256 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:30.382218 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552\": container with ID starting with 8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552 not found: ID does not exist" containerID="8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552" Oct 30 22:31:30.382376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.382267 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552"} err="failed to get container status \"8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552\": rpc error: code = NotFound desc = could not find container \"8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552\": container with ID starting with 8624df432907c115d53cc3a416213596febc901b699688f5bf5e215a7fd31552 not found: ID does not exist" Oct 30 22:31:30.382376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.382294 5607 scope.go:122] "RemoveContainer" containerID="cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287" Oct 30 22:31:30.395385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.395358 5607 scope.go:122] "RemoveContainer" containerID="cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287" Oct 30 22:31:30.395662 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:30.395641 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287\": container with ID starting with cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287 not found: ID does not exist" containerID="cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287" Oct 30 22:31:30.395735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.395670 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287"} err="failed to get container status \"cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287\": rpc error: code = NotFound desc = could not find container \"cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287\": container with ID starting with cbf30dc30006e219e976e650e748023225447a0d4f6c60f366e52ba83604a287 not found: ID does not exist" Oct 30 22:31:30.539483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.539461 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp"] Oct 30 22:31:30.545340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.545321 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c301cf-7957b4776c-qt4cp"] Oct 30 22:31:30.657139 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.657076 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx"] Oct 30 22:31:30.661712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.661680 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c301cf-7889f5fbbf-r7xtx"] Oct 30 22:31:30.872231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.872195 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dcd691-fe9b-4b17-addc-504be4367090" path="/var/lib/kubelet/pods/a7dcd691-fe9b-4b17-addc-504be4367090/volumes" Oct 30 22:31:30.872628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:30.872613 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7f4ee1-1516-4cf6-a581-72db7e851a69" path="/var/lib/kubelet/pods/bc7f4ee1-1516-4cf6-a581-72db7e851a69/volumes" Oct 30 22:31:37.310247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310218 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd"] Oct 30 22:31:37.310811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310797 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a7dcd691-fe9b-4b17-addc-504be4367090" containerName="payload-processing" Oct 30 22:31:37.310867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310812 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dcd691-fe9b-4b17-addc-504be4367090" containerName="payload-processing" Oct 30 22:31:37.310867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310838 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bc7f4ee1-1516-4cf6-a581-72db7e851a69" containerName="payload-pre-processing" Oct 30 22:31:37.310867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310843 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7f4ee1-1516-4cf6-a581-72db7e851a69" containerName="payload-pre-processing" Oct 30 22:31:37.310989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310937 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dcd691-fe9b-4b17-addc-504be4367090" containerName="payload-processing" Oct 30 22:31:37.310989 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.310965 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7f4ee1-1516-4cf6-a581-72db7e851a69" containerName="payload-pre-processing" Oct 30 22:31:37.316230 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.316212 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.370582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.370559 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-fc6466-serving-cert\"" Oct 30 22:31:37.424458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.424428 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd"] Oct 30 22:31:37.464723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.464697 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.464867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.464815 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.464914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.464885 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.465072 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.465058 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qbr\" (UniqueName: \"kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.566025 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.565902 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.566222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.566028 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.566222 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:37.566054 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-serving-cert: secret "maas-api-e2e-shared-a-w3-fc6466-serving-cert" not found Oct 30 22:31:37.566222 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:37.566113 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls podName:69f559a6-a076-4147-8069-694ec06a8575 nodeName:}" failed. No retries permitted until 2026-10-30 22:31:38.066096915 +0000 UTC m=+2677.466653026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls") pod "maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" (UID: "69f559a6-a076-4147-8069-694ec06a8575") : secret "maas-api-e2e-shared-a-w3-fc6466-serving-cert" not found Oct 30 22:31:37.566222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.566059 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.566477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.566428 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qbr\" (UniqueName: \"kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.566886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.566862 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.568929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.568906 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:37.628615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:37.628589 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qbr\" (UniqueName: \"kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:38.072058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:38.072023 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:38.074474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:38.074451 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:38.229944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:38.229913 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:38.603867 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:31:38.603822 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f559a6_a076_4147_8069_694ec06a8575.slice/crio-df228c57fb3377b8bcf73d5ca58a903f888956010a1e32d6cd73195e159b9ef8 WatchSource:0}: Error finding container df228c57fb3377b8bcf73d5ca58a903f888956010a1e32d6cd73195e159b9ef8: Status 404 returned error can't find the container with id df228c57fb3377b8bcf73d5ca58a903f888956010a1e32d6cd73195e159b9ef8 Oct 30 22:31:38.607209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:38.607183 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd"] Oct 30 22:31:39.431465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:39.431426 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" event={"ID":"69f559a6-a076-4147-8069-694ec06a8575","Type":"ContainerStarted","Data":"df228c57fb3377b8bcf73d5ca58a903f888956010a1e32d6cd73195e159b9ef8"} Oct 30 22:31:40.440421 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:40.440381 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" event={"ID":"69f559a6-a076-4147-8069-694ec06a8575","Type":"ContainerStarted","Data":"f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1"} Oct 30 22:31:40.440887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:40.440579 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:40.562602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:40.562538 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" podStartSLOduration=2.538380495 podStartE2EDuration="3.562520117s" totalImagesPullingTime="1.024139622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:40.559755326 +0000 UTC m=+2679.960311447" watchObservedRunningTime="2026-10-30 22:31:40.562520117 +0000 UTC m=+2679.963076239" Oct 30 22:31:46.452978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:31:46.452918 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:31:52.864618 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:31:52.864577 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:32:01.268868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:01.268831 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:32:01.269548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:01.269531 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:32:20.920150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:20.920068 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r"] Oct 30 22:32:20.926379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:20.926352 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:20.982834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:20.982802 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r"] Oct 30 22:32:21.030081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.030039 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-37e79b\"" Oct 30 22:32:21.030650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.030616 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-37e79b-dockercfg-cs9b2\"" Oct 30 22:32:21.058646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.058608 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52tg\" (UniqueName: \"kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg\") pod \"payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.058869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.058852 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.092272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.092244 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5"] Oct 30 22:32:21.096541 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.096521 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.154001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.153979 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5"] Oct 30 22:32:21.160512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.160485 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.160654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.160604 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52tg\" (UniqueName: \"kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg\") pod \"payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.161433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.161411 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.226183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.226154 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52tg\" (UniqueName: \"kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg\") pod \"payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.262009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.261985 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.262159 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.262145 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6j5\" (UniqueName: \"kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5\") pod \"payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.268036 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.268011 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.363542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.363495 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6j5\" (UniqueName: \"kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5\") pod \"payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.364571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.364167 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.365186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.365164 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.426869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.426835 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6j5\" (UniqueName: \"kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5\") pod \"payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.643272 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:32:21.643230 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810d67a6_711d_44c7_a72f_5b6e1b405026.slice/crio-1538067fbeecff1a67ab1e7af602482621fb5ab38fed49a0c7f3ee616f0b7735 WatchSource:0}: Error finding container 1538067fbeecff1a67ab1e7af602482621fb5ab38fed49a0c7f3ee616f0b7735: Status 404 returned error can't find the container with id 1538067fbeecff1a67ab1e7af602482621fb5ab38fed49a0c7f3ee616f0b7735 Oct 30 22:32:21.644284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.644259 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r"] Oct 30 22:32:21.710726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.710703 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:21.764487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.764435 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" event={"ID":"810d67a6-711d-44c7-a72f-5b6e1b405026","Type":"ContainerStarted","Data":"e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487"} Oct 30 22:32:21.764602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.764505 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" event={"ID":"810d67a6-711d-44c7-a72f-5b6e1b405026","Type":"ContainerStarted","Data":"1538067fbeecff1a67ab1e7af602482621fb5ab38fed49a0c7f3ee616f0b7735"} Oct 30 22:32:21.765328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.765072 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:21.885525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:21.885475 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" podStartSLOduration=1.885456945 podStartE2EDuration="1.885456945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:21.883639059 +0000 UTC m=+2721.284195195" watchObservedRunningTime="2026-10-30 22:32:21.885456945 +0000 UTC m=+2721.286013062" Oct 30 22:32:22.082121 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:32:22.082090 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59dd0a1_3355_4607_aecc_d959da666b1a.slice/crio-9504dd9120d069b09b621705bad08334484806d8dae19162c98fe2a97c2dba4d WatchSource:0}: Error finding container 9504dd9120d069b09b621705bad08334484806d8dae19162c98fe2a97c2dba4d: Status 404 returned error can't find the container with id 9504dd9120d069b09b621705bad08334484806d8dae19162c98fe2a97c2dba4d Oct 30 22:32:22.085409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:22.085390 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5"] Oct 30 22:32:22.783673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:22.783618 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" event={"ID":"a59dd0a1-3355-4607-aecc-d959da666b1a","Type":"ContainerStarted","Data":"3f2a8a8186adfd44534679e5c45733e14d7c49b9ff56724c21f95a2248af21f6"} Oct 30 22:32:22.783906 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:22.783707 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" event={"ID":"a59dd0a1-3355-4607-aecc-d959da666b1a","Type":"ContainerStarted","Data":"9504dd9120d069b09b621705bad08334484806d8dae19162c98fe2a97c2dba4d"} Oct 30 22:32:22.866141 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:32:22.866108 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:32:22.904341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:22.904291 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" podStartSLOduration=1.904276536 podStartE2EDuration="1.904276536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:22.902822924 +0000 UTC m=+2722.303379048" watchObservedRunningTime="2026-10-30 22:32:22.904276536 +0000 UTC m=+2722.304832654" Oct 30 22:32:23.790226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:23.790189 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:25.895803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:25.895767 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w"] Oct 30 22:32:25.896263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:25.895978 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" containerID="cri-o://5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7" gracePeriod=30 Oct 30 22:32:30.554926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:30.554884 5607 patch_prober.go:29] interesting pod/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:30.555371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:30.554992 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:31.169227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.169170 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:32:31.209560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209532 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.209695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209571 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.209695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209603 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.209695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209651 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.209856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209705 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.209856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209754 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.209856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209818 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.210080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209873 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.210080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209903 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token\") pod \"2064e2f8-2156-47bc-9601-82874abb8df7\" (UID: \"2064e2f8-2156-47bc-9601-82874abb8df7\") " Oct 30 22:32:31.210080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.209938 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:31.210260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210073 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.210260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210166 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.210260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210239 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.210260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210252 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.210761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210735 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.210902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210766 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.210902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210781 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2064e2f8-2156-47bc-9601-82874abb8df7-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.210902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210795 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.210902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.210809 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.212112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.212090 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:31.212252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.212233 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:31.212395 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.212379 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "kube-api-access-njvz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:31.212465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.212446 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token" pod "2064e2f8-2156-47bc-9601-82874abb8df7" (UID: "2064e2f8-2156-47bc-9601-82874abb8df7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:31.311608 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.311577 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2064e2f8-2156-47bc-9601-82874abb8df7-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.311608 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.311603 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2064e2f8-2156-47bc-9601-82874abb8df7-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.311608 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.311615 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvz6\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-kube-api-access-njvz6\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.311841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.311626 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2064e2f8-2156-47bc-9601-82874abb8df7-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:31.845700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.845663 5607 generic.go:356] "Generic (PLEG): container finished" podID="2064e2f8-2156-47bc-9601-82874abb8df7" containerID="5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7" exitCode=0 Oct 30 22:32:31.846247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.845723 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" Oct 30 22:32:31.846247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.845756 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" event={"ID":"2064e2f8-2156-47bc-9601-82874abb8df7","Type":"ContainerDied","Data":"5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7"} Oct 30 22:32:31.846247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.845830 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w" event={"ID":"2064e2f8-2156-47bc-9601-82874abb8df7","Type":"ContainerDied","Data":"30fdf39c97ce3a1f879eeff5697f67ac0343e83c67475f47239c09d5a6fe16e3"} Oct 30 22:32:31.846247 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.845863 5607 scope.go:122] "RemoveContainer" containerID="5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7" Oct 30 22:32:31.856963 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.856930 5607 scope.go:122] "RemoveContainer" containerID="7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37" Oct 30 22:32:31.866384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.866365 5607 scope.go:122] "RemoveContainer" containerID="5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7" Oct 30 22:32:31.866691 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:32:31.866669 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7\": container with ID starting with 5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7 not found: ID does not exist" containerID="5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7" Oct 30 22:32:31.866772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.866699 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7"} err="failed to get container status \"5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7\": rpc error: code = NotFound desc = could not find container \"5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7\": container with ID starting with 5fc38be43b3f52c433fcd0217a4b191f28e0b6da742c2e87fa63ce98460ceca7 not found: ID does not exist" Oct 30 22:32:31.866772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.866721 5607 scope.go:122] "RemoveContainer" containerID="7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37" Oct 30 22:32:31.866988 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:32:31.866942 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37\": container with ID starting with 7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37 not found: ID does not exist" containerID="7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37" Oct 30 22:32:31.867046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:31.866993 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37"} err="failed to get container status \"7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37\": rpc error: code = NotFound desc = could not find container \"7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37\": container with ID starting with 7154a88ee7befdf7d7b7664ab77fbc61b2144b1cf1b59d69b2ec6b4ae39afe37 not found: ID does not exist" Oct 30 22:32:32.016113 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:32.016080 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w"] Oct 30 22:32:32.067151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:32.067116 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c301cf-openshift-default-849d8fb757-99z6w"] Oct 30 22:32:32.786855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:32.786829 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:32:32.870495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:32.870466 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" path="/var/lib/kubelet/pods/2064e2f8-2156-47bc-9601-82874abb8df7/volumes" Oct 30 22:32:34.801113 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:34.801087 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:32:35.931612 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.931579 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st"] Oct 30 22:32:35.934943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.933069 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" Oct 30 22:32:35.934943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.933093 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" Oct 30 22:32:35.934943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.933136 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" Oct 30 22:32:35.934943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.933143 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" Oct 30 22:32:35.934943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.933362 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" Oct 30 22:32:35.940797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:35.940771 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.007947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.007884 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st"] Oct 30 22:32:36.039599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.039491 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-3b392d\"" Oct 30 22:32:36.039599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.039535 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-3b392d-dockercfg-2ljwh\"" Oct 30 22:32:36.072803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.072771 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.073061 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.073035 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzhx\" (UniqueName: \"kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx\") pod \"payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.099177 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.099143 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx"] Oct 30 22:32:36.099873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.099860 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2064e2f8-2156-47bc-9601-82874abb8df7" containerName="istio-proxy" Oct 30 22:32:36.103311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.103295 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.161626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.161593 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx"] Oct 30 22:32:36.174225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.174197 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx462\" (UniqueName: \"kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462\") pod \"payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.174381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.174264 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.174381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.174349 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzhx\" (UniqueName: \"kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx\") pod \"payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.174455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.174445 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.175242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.175221 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.237101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.237068 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzhx\" (UniqueName: \"kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx\") pod \"payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.275310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.275279 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx462\" (UniqueName: \"kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462\") pod \"payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.275477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.275341 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.276106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.276086 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.288449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.288408 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:36.338456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.338418 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx462\" (UniqueName: \"kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462\") pod \"payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.417813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.417770 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:36.664023 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:32:36.663990 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571973ac_dd21_4647_8eda_a9d56f5e9d73.slice/crio-3fee49dd7c7088798ab7c8adc10d60ac530934abae0ac677ec408bed658df000 WatchSource:0}: Error finding container 3fee49dd7c7088798ab7c8adc10d60ac530934abae0ac677ec408bed658df000: Status 404 returned error can't find the container with id 3fee49dd7c7088798ab7c8adc10d60ac530934abae0ac677ec408bed658df000 Oct 30 22:32:36.666636 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.666609 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st"] Oct 30 22:32:36.802699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.802660 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx"] Oct 30 22:32:36.880872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.880760 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" event={"ID":"78ca2c17-2aa9-4b52-b9da-356d099811b9","Type":"ContainerStarted","Data":"1cb0e232335ff95720c8fa9715d63e4c6c2102d4b197bf891879a7fed19cb58f"} Oct 30 22:32:36.882465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.882438 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" event={"ID":"571973ac-dd21-4647-8eda-a9d56f5e9d73","Type":"ContainerStarted","Data":"88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c"} Oct 30 22:32:36.882602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.882478 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" event={"ID":"571973ac-dd21-4647-8eda-a9d56f5e9d73","Type":"ContainerStarted","Data":"3fee49dd7c7088798ab7c8adc10d60ac530934abae0ac677ec408bed658df000"} Oct 30 22:32:36.882602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:36.882502 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:37.005525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:37.005454 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" podStartSLOduration=2.005435609 podStartE2EDuration="2.005435609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:37.000760001 +0000 UTC m=+2736.401316148" watchObservedRunningTime="2026-10-30 22:32:37.005435609 +0000 UTC m=+2736.405991730" Oct 30 22:32:37.900183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:37.900139 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" event={"ID":"78ca2c17-2aa9-4b52-b9da-356d099811b9","Type":"ContainerStarted","Data":"6b514eb1950b897db9bbaed005f0098ba083971417fd45294751e231357d7d42"} Oct 30 22:32:38.028987 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:38.028900 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" podStartSLOduration=2.028886758 podStartE2EDuration="2.028886758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:38.026625143 +0000 UTC m=+2737.427181261" watchObservedRunningTime="2026-10-30 22:32:38.028886758 +0000 UTC m=+2737.429442878" Oct 30 22:32:38.907522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:38.907493 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:39.788599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:39.788565 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t"] Oct 30 22:32:39.789075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:39.788764 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" containerID="cri-o://66cdad24981357135db4e34a60262935921293bf40d00360a66cd7ffd2f0c8e8" gracePeriod=30 Oct 30 22:32:39.839219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:39.839184 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h"] Oct 30 22:32:39.839425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:39.839404 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" containerID="cri-o://e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9" gracePeriod=30 Oct 30 22:32:40.373571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:40.373531 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-h2z4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:40.373778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:40.373597 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:40.563779 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:40.563744 5607 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-8rl4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:40.563945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:40.563796 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:44.953426 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:44.953392 5607 generic.go:356] "Generic (PLEG): container finished" podID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerID="66cdad24981357135db4e34a60262935921293bf40d00360a66cd7ffd2f0c8e8" exitCode=0 Oct 30 22:32:44.954014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:44.953476 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" event={"ID":"6274436d-a97f-4298-9edf-1b1a68fc1332","Type":"ContainerDied","Data":"66cdad24981357135db4e34a60262935921293bf40d00360a66cd7ffd2f0c8e8"} Oct 30 22:32:45.115848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.115818 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:32:45.165787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.165764 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:32:45.303212 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303173 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.303413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303262 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.303413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303298 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.303413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303324 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.303413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303364 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.303413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303408 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303443 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303486 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303536 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303580 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303610 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303689 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:45.303719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303702 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303733 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303784 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303822 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303852 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data\") pod \"6274436d-a97f-4298-9edf-1b1a68fc1332\" (UID: \"6274436d-a97f-4298-9edf-1b1a68fc1332\") " Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303899 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.303947 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.304355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.304082 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs\") pod \"47a2e457-26de-4744-b8a7-39f952cb4cae\" (UID: \"47a2e457-26de-4744-b8a7-39f952cb4cae\") " Oct 30 22:32:45.304757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.304622 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.304757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.304642 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47a2e457-26de-4744-b8a7-39f952cb4cae-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.304757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.304665 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.304925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.304877 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.305010 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.304934 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:45.305673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.305102 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.305673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.305259 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.305673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.305313 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.305673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.305492 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.305673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.305558 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.306492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.306433 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:45.306692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.306661 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:45.307492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.307459 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:45.308034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.308003 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:45.308281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.308258 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.308437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.308410 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z" pod "47a2e457-26de-4744-b8a7-39f952cb4cae" (UID: "47a2e457-26de-4744-b8a7-39f952cb4cae"). InnerVolumeSpecName "kube-api-access-7p96z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:45.308673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.308647 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "kube-api-access-kxwc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:45.309069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.309049 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy" pod "6274436d-a97f-4298-9edf-1b1a68fc1332" (UID: "6274436d-a97f-4298-9edf-1b1a68fc1332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408514 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408550 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408567 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408581 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408596 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408611 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408624 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408639 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408653 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408667 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408682 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6274436d-a97f-4298-9edf-1b1a68fc1332-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408697 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwc6\" (UniqueName: \"kubernetes.io/projected/6274436d-a97f-4298-9edf-1b1a68fc1332-kube-api-access-kxwc6\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408712 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6274436d-a97f-4298-9edf-1b1a68fc1332-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408727 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p96z\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-kube-api-access-7p96z\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408742 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47a2e457-26de-4744-b8a7-39f952cb4cae-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.408756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.408756 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47a2e457-26de-4744-b8a7-39f952cb4cae-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:45.961830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.961798 5607 generic.go:356] "Generic (PLEG): container finished" podID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerID="e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9" exitCode=0 Oct 30 22:32:45.962266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.961843 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" Oct 30 22:32:45.962266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.961843 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" event={"ID":"47a2e457-26de-4744-b8a7-39f952cb4cae","Type":"ContainerDied","Data":"e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9"} Oct 30 22:32:45.962266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.962000 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h" event={"ID":"47a2e457-26de-4744-b8a7-39f952cb4cae","Type":"ContainerDied","Data":"ccb37d5ff064e0034cb9e924902b45fdef7951b6106cfbb09b2c04d768d93c0f"} Oct 30 22:32:45.962266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.962033 5607 scope.go:122] "RemoveContainer" containerID="e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9" Oct 30 22:32:45.963868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.963838 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" Oct 30 22:32:45.963868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.963842 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t" event={"ID":"6274436d-a97f-4298-9edf-1b1a68fc1332","Type":"ContainerDied","Data":"2e9f6ba877badb991611019e1296856ffe08a30d6af570e0642ebccf17d75814"} Oct 30 22:32:45.973724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.973703 5607 scope.go:122] "RemoveContainer" containerID="e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9" Oct 30 22:32:45.974132 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:32:45.974103 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9\": container with ID starting with e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9 not found: ID does not exist" containerID="e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9" Oct 30 22:32:45.974217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.974143 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9"} err="failed to get container status \"e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9\": rpc error: code = NotFound desc = could not find container \"e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9\": container with ID starting with e80a8366faf6f25736d328e607b3baf7599329670c0fc03e3df0b905239a8ae9 not found: ID does not exist" Oct 30 22:32:45.974217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:45.974160 5607 scope.go:122] "RemoveContainer" containerID="66cdad24981357135db4e34a60262935921293bf40d00360a66cd7ffd2f0c8e8" Oct 30 22:32:46.131638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:46.131578 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t"] Oct 30 22:32:46.182691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:46.182670 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-8rl4t"] Oct 30 22:32:46.294900 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:46.294879 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h"] Oct 30 22:32:46.298653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:46.298631 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-h2z4h"] Oct 30 22:32:46.872258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:46.872215 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" path="/var/lib/kubelet/pods/47a2e457-26de-4744-b8a7-39f952cb4cae/volumes" Oct 30 22:32:46.872786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:46.872762 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" path="/var/lib/kubelet/pods/6274436d-a97f-4298-9edf-1b1a68fc1332/volumes" Oct 30 22:32:47.382289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.382259 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n"] Oct 30 22:32:47.382880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.382867 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" Oct 30 22:32:47.382930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.382881 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" Oct 30 22:32:47.382930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.382912 5607 cpu_manager.go:414] "containerMap: removing container" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" Oct 30 22:32:47.382930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.382917 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" Oct 30 22:32:47.383062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.383020 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a2e457-26de-4744-b8a7-39f952cb4cae" containerName="istio-proxy" Oct 30 22:32:47.383062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.383033 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6274436d-a97f-4298-9edf-1b1a68fc1332" containerName="istio-proxy" Oct 30 22:32:47.388269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.388249 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.431871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.431824 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrdr\" (UniqueName: \"kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr\") pod \"payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.432083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.431891 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.442343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.442313 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-435923\"" Oct 30 22:32:47.449016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.448984 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n"] Oct 30 22:32:47.493474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.493439 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-435923-dockercfg-5lrvg\"" Oct 30 22:32:47.532815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.532778 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrdr\" (UniqueName: \"kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr\") pod \"payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.533038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.532845 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.533573 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.533557 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.556144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.556117 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77"] Oct 30 22:32:47.560436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.560417 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.596295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.596264 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrdr\" (UniqueName: \"kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr\") pod \"payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.622244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.622208 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77"] Oct 30 22:32:47.634412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.634332 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.634412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.634399 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gzt\" (UniqueName: \"kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt\") pod \"payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.706408 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.706375 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:47.735805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.735774 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.735926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.735860 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gzt\" (UniqueName: \"kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt\") pod \"payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.736808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.736783 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.799647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.799621 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gzt\" (UniqueName: \"kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt\") pod \"payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.874677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.874647 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:47.904180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:47.904160 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:32:48.127876 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:32:48.127847 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7899e2f7_a654_490b_aa02_af4124d6b43b.slice/crio-d410270449d883474cd047b1cb59133efd4e9402b7bef89aec9beabf976687f5 WatchSource:0}: Error finding container d410270449d883474cd047b1cb59133efd4e9402b7bef89aec9beabf976687f5: Status 404 returned error can't find the container with id d410270449d883474cd047b1cb59133efd4e9402b7bef89aec9beabf976687f5 Oct 30 22:32:48.130685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:48.130661 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n"] Oct 30 22:32:48.249412 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:32:48.249381 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3dd339_b978_4b7a_998b_587e00c82bca.slice/crio-dda4d53e55e43734ae84dd7613763d5003d42c3ad19c7b738f2e24f3abe1f343 WatchSource:0}: Error finding container dda4d53e55e43734ae84dd7613763d5003d42c3ad19c7b738f2e24f3abe1f343: Status 404 returned error can't find the container with id dda4d53e55e43734ae84dd7613763d5003d42c3ad19c7b738f2e24f3abe1f343 Oct 30 22:32:48.253038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:48.253018 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77"] Oct 30 22:32:49.011662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.011617 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" event={"ID":"7899e2f7-a654-490b-aa02-af4124d6b43b","Type":"ContainerStarted","Data":"284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316"} Oct 30 22:32:49.012136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.011677 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" event={"ID":"7899e2f7-a654-490b-aa02-af4124d6b43b","Type":"ContainerStarted","Data":"d410270449d883474cd047b1cb59133efd4e9402b7bef89aec9beabf976687f5"} Oct 30 22:32:49.012136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.011799 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:32:49.013294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.013268 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" event={"ID":"6c3dd339-b978-4b7a-998b-587e00c82bca","Type":"ContainerStarted","Data":"53de3ccb94a6a5e574606e537c8c29d423d37450376f958ee855afa1f8adfda0"} Oct 30 22:32:49.013401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.013319 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:32:49.013401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.013337 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" event={"ID":"6c3dd339-b978-4b7a-998b-587e00c82bca","Type":"ContainerStarted","Data":"dda4d53e55e43734ae84dd7613763d5003d42c3ad19c7b738f2e24f3abe1f343"} Oct 30 22:32:49.131697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.131654 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" podStartSLOduration=2.131640726 podStartE2EDuration="2.131640726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:49.128911016 +0000 UTC m=+2748.529467139" watchObservedRunningTime="2026-10-30 22:32:49.131640726 +0000 UTC m=+2748.532196848" Oct 30 22:32:49.248586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.248547 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" podStartSLOduration=2.248533546 podStartE2EDuration="2.248533546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:49.246917541 +0000 UTC m=+2748.647473653" watchObservedRunningTime="2026-10-30 22:32:49.248533546 +0000 UTC m=+2748.649089692" Oct 30 22:32:49.918461 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:49.918420 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:32:51.055978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:51.055907 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn"] Oct 30 22:32:51.106800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:51.106774 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ea3efd7-kkvbn"] Oct 30 22:32:52.864391 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:32:52.864345 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:32:52.871039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:32:52.871011 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d552e1-110d-432e-8ad8-e56a669a8076" path="/var/lib/kubelet/pods/64d552e1-110d-432e-8ad8-e56a669a8076/volumes" Oct 30 22:33:00.022904 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:00.022870 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:33:00.023433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:00.023197 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:33:01.088450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.088414 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g"] Oct 30 22:33:01.092792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.092771 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.177215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.177190 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.177353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.177242 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdm88\" (UniqueName: \"kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.177353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.177313 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.177511 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.177483 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.199039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.199017 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-68f80c-serving-cert\"" Oct 30 22:33:01.202034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.202013 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g"] Oct 30 22:33:01.278402 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.278377 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.278560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.278435 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdm88\" (UniqueName: \"kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.278560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.278462 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.278560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.278542 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.278744 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:01.278649 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-serving-cert: secret "maas-api-e2e-shared-b-w2-68f80c-serving-cert" not found Oct 30 22:33:01.278744 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:01.278730 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls podName:2216473b-6bc3-4d55-b118-9bc2de085859 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:01.778708218 +0000 UTC m=+2761.179264320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls") pod "maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" (UID: "2216473b-6bc3-4d55-b118-9bc2de085859") : secret "maas-api-e2e-shared-b-w2-68f80c-serving-cert" not found Oct 30 22:33:01.279290 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.279218 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.280932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.280912 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.341828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.341704 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdm88\" (UniqueName: \"kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.784495 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.784450 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:01.787056 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:01.787024 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:02.007889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:02.007861 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:02.392330 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:33:02.392300 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2216473b_6bc3_4d55_b118_9bc2de085859.slice/crio-5e089462b36bcce57e111c00e14c3239856448cccfffca77954f59a9d11d3a3b WatchSource:0}: Error finding container 5e089462b36bcce57e111c00e14c3239856448cccfffca77954f59a9d11d3a3b: Status 404 returned error can't find the container with id 5e089462b36bcce57e111c00e14c3239856448cccfffca77954f59a9d11d3a3b Oct 30 22:33:02.393352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:02.393331 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g"] Oct 30 22:33:02.393786 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:02.393768 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:33:03.115444 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:03.115414 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" event={"ID":"2216473b-6bc3-4d55-b118-9bc2de085859","Type":"ContainerStarted","Data":"5e089462b36bcce57e111c00e14c3239856448cccfffca77954f59a9d11d3a3b"} Oct 30 22:33:04.123838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:04.123798 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" event={"ID":"2216473b-6bc3-4d55-b118-9bc2de085859","Type":"ContainerStarted","Data":"23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5"} Oct 30 22:33:04.124341 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:04.123898 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:04.256685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:04.253255 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" podStartSLOduration=2.395024553 podStartE2EDuration="3.253231807s" totalImagesPullingTime="858.207254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:04.24901132 +0000 UTC m=+2763.649567447" watchObservedRunningTime="2026-10-30 22:33:04.253231807 +0000 UTC m=+2763.653787929" Oct 30 22:33:10.139885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:10.139856 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:33:15.076611 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.076571 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z"] Oct 30 22:33:15.082083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.082056 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.188682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.188658 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-37e79b-serving-cert\"" Oct 30 22:33:15.191726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.191702 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z"] Oct 30 22:33:15.239456 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.239431 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.239624 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.239478 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jfj\" (UniqueName: \"kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.239704 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.239644 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.239756 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.239742 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.341679 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.341892 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.341989 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jfj\" (UniqueName: \"kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.342112 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.343038 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:15.343172 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-serving-cert: secret "maas-api-e2e-shared-b-w4-37e79b-serving-cert" not found Oct 30 22:33:15.343881 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:15.343250 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls podName:a4c5d315-965e-4315-a243-0af5c701cfd2 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:15.843229551 +0000 UTC m=+2775.243785679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls") pod "maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" (UID: "a4c5d315-965e-4315-a243-0af5c701cfd2") : secret "maas-api-e2e-shared-b-w4-37e79b-serving-cert" not found Oct 30 22:33:15.376605 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.376569 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.408482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.408454 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jfj\" (UniqueName: \"kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.847378 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.847345 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.849767 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.849741 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:15.996604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:15.996581 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:16.379815 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:33:16.379784 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c5d315_965e_4315_a243_0af5c701cfd2.slice/crio-32c8fc4945865985ad7ddf807ab54c1fc39501733bc52b9775c90a75ea8a806a WatchSource:0}: Error finding container 32c8fc4945865985ad7ddf807ab54c1fc39501733bc52b9775c90a75ea8a806a: Status 404 returned error can't find the container with id 32c8fc4945865985ad7ddf807ab54c1fc39501733bc52b9775c90a75ea8a806a Oct 30 22:33:16.383450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:16.383424 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z"] Oct 30 22:33:17.222210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:17.222175 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" event={"ID":"a4c5d315-965e-4315-a243-0af5c701cfd2","Type":"ContainerStarted","Data":"32c8fc4945865985ad7ddf807ab54c1fc39501733bc52b9775c90a75ea8a806a"} Oct 30 22:33:18.234060 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:18.234004 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" event={"ID":"a4c5d315-965e-4315-a243-0af5c701cfd2","Type":"ContainerStarted","Data":"0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498"} Oct 30 22:33:18.234539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:18.234234 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:18.354331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:18.354288 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" podStartSLOduration=2.461952683 podStartE2EDuration="3.35427576s" totalImagesPullingTime="892.323077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:18.353567134 +0000 UTC m=+2777.754123254" watchObservedRunningTime="2026-10-30 22:33:18.35427576 +0000 UTC m=+2777.754831881" Oct 30 22:33:22.864406 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:22.864372 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:33:24.247793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:24.247759 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:33:32.071583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:32.071548 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b"] Oct 30 22:33:32.183410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:32.183381 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52"] Oct 30 22:33:34.328694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.328663 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9"] Oct 30 22:33:34.333182 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.333165 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.440350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.440319 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3b392d-serving-cert\"" Oct 30 22:33:34.443500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.443479 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9"] Oct 30 22:33:34.458397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.458365 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8wv\" (UniqueName: \"kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.458613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.458450 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.458613 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.458587 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.458749 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.458671 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.559603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.559572 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.559790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.559643 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.559790 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.559698 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8wv\" (UniqueName: \"kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.559927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.559840 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.560095 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:34.559972 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-serving-cert: secret "maas-api-e2e-shared-a-w0-3b392d-serving-cert" not found Oct 30 22:33:34.560095 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:34.560038 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls podName:c397a2bb-d114-4394-ab83-5b3c0ba77994 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:35.060020886 +0000 UTC m=+2794.460576986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls") pod "maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" (UID: "c397a2bb-d114-4394-ab83-5b3c0ba77994") : secret "maas-api-e2e-shared-a-w0-3b392d-serving-cert" not found Oct 30 22:33:34.560896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.560877 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.562475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.562443 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:34.623236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:34.623156 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8wv\" (UniqueName: \"kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:35.065776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:35.065737 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:35.068365 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:35.068345 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:35.247151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:35.247118 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:35.625751 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:33:35.625723 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc397a2bb_d114_4394_ab83_5b3c0ba77994.slice/crio-552f5c2c002d22b2600af716b567ed432e94956d922657251303634afb83302c WatchSource:0}: Error finding container 552f5c2c002d22b2600af716b567ed432e94956d922657251303634afb83302c: Status 404 returned error can't find the container with id 552f5c2c002d22b2600af716b567ed432e94956d922657251303634afb83302c Oct 30 22:33:35.631253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:35.631229 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9"] Oct 30 22:33:36.365106 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:36.365067 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" event={"ID":"c397a2bb-d114-4394-ab83-5b3c0ba77994","Type":"ContainerStarted","Data":"552f5c2c002d22b2600af716b567ed432e94956d922657251303634afb83302c"} Oct 30 22:33:37.072591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:37.072551 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" podUID="735721fb-43c5-4c59-833b-75666cdb2451" containerName="payload-processing" containerID="cri-o://e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3" gracePeriod=25 Oct 30 22:33:37.185443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:37.185367 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerName="payload-pre-processing" containerID="cri-o://c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e" gracePeriod=25 Oct 30 22:33:37.375071 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:37.374988 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" event={"ID":"c397a2bb-d114-4394-ab83-5b3c0ba77994","Type":"ContainerStarted","Data":"72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f"} Oct 30 22:33:37.375231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:37.375122 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:37.495943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:37.495898 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" podStartSLOduration=2.568811609 podStartE2EDuration="3.495884604s" totalImagesPullingTime="927.072995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:37.495092978 +0000 UTC m=+2796.895649100" watchObservedRunningTime="2026-10-30 22:33:37.495884604 +0000 UTC m=+2796.896440725" Oct 30 22:33:38.270000 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:38.269940 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:39.275154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:39.275115 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" podUID="735721fb-43c5-4c59-833b-75666cdb2451" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:43.387718 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:43.387646 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:33:48.269715 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:48.269676 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:49.275834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:49.275797 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" podUID="735721fb-43c5-4c59-833b-75666cdb2451" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:33:52.864862 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:52.864823 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:33:57.357261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.357234 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:33:57.430838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.430809 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp\" (UniqueName: \"kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp\") pod \"735721fb-43c5-4c59-833b-75666cdb2451\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " Oct 30 22:33:57.431016 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.430856 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume\") pod \"735721fb-43c5-4c59-833b-75666cdb2451\" (UID: \"735721fb-43c5-4c59-833b-75666cdb2451\") " Oct 30 22:33:57.431263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.431239 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume" pod "735721fb-43c5-4c59-833b-75666cdb2451" (UID: "735721fb-43c5-4c59-833b-75666cdb2451"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:57.431387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.431366 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/735721fb-43c5-4c59-833b-75666cdb2451-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:57.433156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.433136 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp" pod "735721fb-43c5-4c59-833b-75666cdb2451" (UID: "735721fb-43c5-4c59-833b-75666cdb2451"). InnerVolumeSpecName "kube-api-access-pgfrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:57.448354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.448336 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:33:57.508693 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.508665 5607 generic.go:356] "Generic (PLEG): container finished" podID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerID="c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e" exitCode=0 Oct 30 22:33:57.508842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.508716 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" Oct 30 22:33:57.508842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.508756 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" event={"ID":"f7d59d01-d737-4fc8-abba-6a3bca7d12bd","Type":"ContainerDied","Data":"c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e"} Oct 30 22:33:57.508842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.508817 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52" event={"ID":"f7d59d01-d737-4fc8-abba-6a3bca7d12bd","Type":"ContainerDied","Data":"2f9dea9759a9777a17d4bfb3be34e33c3def59bf76615e5275fd0751ed2d8a0f"} Oct 30 22:33:57.509048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.508849 5607 scope.go:122] "RemoveContainer" containerID="c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e" Oct 30 22:33:57.510619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.510520 5607 generic.go:356] "Generic (PLEG): container finished" podID="735721fb-43c5-4c59-833b-75666cdb2451" containerID="e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3" exitCode=0 Oct 30 22:33:57.510619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.510572 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" event={"ID":"735721fb-43c5-4c59-833b-75666cdb2451","Type":"ContainerDied","Data":"e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3"} Oct 30 22:33:57.510619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.510604 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" event={"ID":"735721fb-43c5-4c59-833b-75666cdb2451","Type":"ContainerDied","Data":"36cb4802121a4521384bec96421ebeb3997fd7fdbd744cddf7d4ba0ad6cfa518"} Oct 30 22:33:57.510868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.510660 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b" Oct 30 22:33:57.521206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.521187 5607 scope.go:122] "RemoveContainer" containerID="c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e" Oct 30 22:33:57.521519 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:57.521493 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e\": container with ID starting with c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e not found: ID does not exist" containerID="c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e" Oct 30 22:33:57.521587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.521532 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e"} err="failed to get container status \"c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e\": rpc error: code = NotFound desc = could not find container \"c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e\": container with ID starting with c8bcd10499f30e73d766694496d09ab791d772b9f2e16ca3d6c4ee620d57650e not found: ID does not exist" Oct 30 22:33:57.521587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.521557 5607 scope.go:122] "RemoveContainer" containerID="e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3" Oct 30 22:33:57.531121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.531104 5607 scope.go:122] "RemoveContainer" containerID="e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3" Oct 30 22:33:57.531431 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:33:57.531407 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3\": container with ID starting with e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3 not found: ID does not exist" containerID="e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3" Oct 30 22:33:57.531501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.531439 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3"} err="failed to get container status \"e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3\": rpc error: code = NotFound desc = could not find container \"e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3\": container with ID starting with e1dc2e04cab55629b7ab29101682eaaa2436657c310502134406e231d7ca91f3 not found: ID does not exist" Oct 30 22:33:57.531741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.531725 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h\" (UniqueName: \"kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h\") pod \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " Oct 30 22:33:57.531890 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.531878 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume\") pod \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\" (UID: \"f7d59d01-d737-4fc8-abba-6a3bca7d12bd\") " Oct 30 22:33:57.532292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.532270 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume" pod "f7d59d01-d737-4fc8-abba-6a3bca7d12bd" (UID: "f7d59d01-d737-4fc8-abba-6a3bca7d12bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:57.532556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.532540 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:57.532621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.532562 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgfrp\" (UniqueName: \"kubernetes.io/projected/735721fb-43c5-4c59-833b-75666cdb2451-kube-api-access-pgfrp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:57.534967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.534931 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h" pod "f7d59d01-d737-4fc8-abba-6a3bca7d12bd" (UID: "f7d59d01-d737-4fc8-abba-6a3bca7d12bd"). InnerVolumeSpecName "kube-api-access-wbb5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:57.634291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.634210 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbb5h\" (UniqueName: \"kubernetes.io/projected/f7d59d01-d737-4fc8-abba-6a3bca7d12bd-kube-api-access-wbb5h\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:57.764294 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.764262 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b"] Oct 30 22:33:57.770310 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.770291 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1875b4-785945b489-gx84b"] Oct 30 22:33:57.933582 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.933504 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52"] Oct 30 22:33:57.936543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:57.936521 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1875b4-bf5c6b5c9-jxk52"] Oct 30 22:33:58.870400 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:58.870369 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735721fb-43c5-4c59-833b-75666cdb2451" path="/var/lib/kubelet/pods/735721fb-43c5-4c59-833b-75666cdb2451/volumes" Oct 30 22:33:58.870788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:33:58.870734 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" path="/var/lib/kubelet/pods/f7d59d01-d737-4fc8-abba-6a3bca7d12bd/volumes" Oct 30 22:34:02.402643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:34:02.402612 5607 scope.go:122] "RemoveContainer" containerID="a01e528c315f1ccb101ae9db044c07273776beed9a60952da3cc88174938f5c5" Oct 30 22:34:22.865157 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:34:22.865116 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:34:52.866098 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:34:52.866060 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:35:00.136745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.136706 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc"] Oct 30 22:35:00.137423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.137407 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerName="payload-pre-processing" Oct 30 22:35:00.137490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.137423 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerName="payload-pre-processing" Oct 30 22:35:00.137490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.137452 5607 cpu_manager.go:414] "containerMap: removing container" podUID="735721fb-43c5-4c59-833b-75666cdb2451" containerName="payload-processing" Oct 30 22:35:00.137490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.137456 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735721fb-43c5-4c59-833b-75666cdb2451" containerName="payload-processing" Oct 30 22:35:00.137596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.137579 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d59d01-d737-4fc8-abba-6a3bca7d12bd" containerName="payload-pre-processing" Oct 30 22:35:00.137596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.137592 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="735721fb-43c5-4c59-833b-75666cdb2451" containerName="payload-processing" Oct 30 22:35:00.141157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.141136 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.195048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.195026 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-74bb81\"" Oct 30 22:35:00.195165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.195148 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-74bb81-dockercfg-tb4hq\"" Oct 30 22:35:00.265935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.265868 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrsq\" (UniqueName: \"kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq\") pod \"payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.266518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.266482 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.271664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.271637 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc"] Oct 30 22:35:00.307163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.307135 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd"] Oct 30 22:35:00.311437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.311419 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.367350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.367322 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.367508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.367491 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrsq\" (UniqueName: \"kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq\") pod \"payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.368143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.368119 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.372052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.372028 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd"] Oct 30 22:35:00.431361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.431297 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrsq\" (UniqueName: \"kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq\") pod \"payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.455077 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.455051 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:00.468728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.468701 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.468825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.468774 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsb25\" (UniqueName: \"kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25\") pod \"payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.569986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.569919 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.570142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.570083 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsb25\" (UniqueName: \"kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25\") pod \"payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.570754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.570736 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.633167 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.633142 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsb25\" (UniqueName: \"kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25\") pod \"payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.826490 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:35:00.826455 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b02ea1_6c5f_44dc_89de_1d7e2241c7d1.slice/crio-457d7cdc5b40f774d1e461374967762e1bf3d21e0cffb84ef091303535ee0ffe WatchSource:0}: Error finding container 457d7cdc5b40f774d1e461374967762e1bf3d21e0cffb84ef091303535ee0ffe: Status 404 returned error can't find the container with id 457d7cdc5b40f774d1e461374967762e1bf3d21e0cffb84ef091303535ee0ffe Oct 30 22:35:00.829680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.829655 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc"] Oct 30 22:35:00.927848 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.927826 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:00.947728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.947695 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" event={"ID":"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1","Type":"ContainerStarted","Data":"728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3"} Oct 30 22:35:00.947854 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.947768 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" event={"ID":"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1","Type":"ContainerStarted","Data":"457d7cdc5b40f774d1e461374967762e1bf3d21e0cffb84ef091303535ee0ffe"} Oct 30 22:35:00.947854 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:00.947807 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:01.071363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:01.071308 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" podStartSLOduration=1.071292426 podStartE2EDuration="1.071292426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:01.070364251 +0000 UTC m=+2880.470920372" watchObservedRunningTime="2026-10-30 22:35:01.071292426 +0000 UTC m=+2880.471848548" Oct 30 22:35:01.301873 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:35:01.301826 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0f2b62_a071_4893_9003_da224c78556c.slice/crio-d032873e30919dfa3f60cae0d8af87be0f2af6876dd4a03f5956f0f2b880f8d3 WatchSource:0}: Error finding container d032873e30919dfa3f60cae0d8af87be0f2af6876dd4a03f5956f0f2b880f8d3: Status 404 returned error can't find the container with id d032873e30919dfa3f60cae0d8af87be0f2af6876dd4a03f5956f0f2b880f8d3 Oct 30 22:35:01.305587 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:01.305564 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd"] Oct 30 22:35:01.959964 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:01.959928 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" event={"ID":"eb0f2b62-a071-4893-9003-da224c78556c","Type":"ContainerStarted","Data":"eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e"} Oct 30 22:35:01.960196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:01.960014 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" event={"ID":"eb0f2b62-a071-4893-9003-da224c78556c","Type":"ContainerStarted","Data":"d032873e30919dfa3f60cae0d8af87be0f2af6876dd4a03f5956f0f2b880f8d3"} Oct 30 22:35:01.960196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:01.960160 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:02.080161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:02.080119 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" podStartSLOduration=2.080105644 podStartE2EDuration="2.080105644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:02.078238254 +0000 UTC m=+2881.478794375" watchObservedRunningTime="2026-10-30 22:35:02.080105644 +0000 UTC m=+2881.480661765" Oct 30 22:35:11.963009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:11.962931 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:35:12.970084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:12.970058 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:35:22.863670 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:35:22.863634 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:35:36.275680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.275643 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865"] Oct 30 22:35:36.276222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.275839 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" podUID="d7b8e021-a67e-4d55-b4a7-b5863c65a981" containerName="maas-api" containerID="cri-o://1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959" gracePeriod=30 Oct 30 22:35:36.571888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.571857 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:35:36.755070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.755041 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\") pod \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " Oct 30 22:35:36.755296 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.755114 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82\" (UniqueName: \"kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82\") pod \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " Oct 30 22:35:36.755390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.755299 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs\") pod \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " Oct 30 22:35:36.755461 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.755437 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca\") pod \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\" (UID: \"d7b8e021-a67e-4d55-b4a7-b5863c65a981\") " Oct 30 22:35:36.755868 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.755831 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca" pod "d7b8e021-a67e-4d55-b4a7-b5863c65a981" (UID: "d7b8e021-a67e-4d55-b4a7-b5863c65a981"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:36.756231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.756210 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8e021-a67e-4d55-b4a7-b5863c65a981-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.757500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.757474 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82" pod "d7b8e021-a67e-4d55-b4a7-b5863c65a981" (UID: "d7b8e021-a67e-4d55-b4a7-b5863c65a981"). InnerVolumeSpecName "kube-api-access-s8h82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.757500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.757491 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs" pod "d7b8e021-a67e-4d55-b4a7-b5863c65a981" (UID: "d7b8e021-a67e-4d55-b4a7-b5863c65a981"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:36.757685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.757516 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls" pod "d7b8e021-a67e-4d55-b4a7-b5863c65a981" (UID: "d7b8e021-a67e-4d55-b4a7-b5863c65a981"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:36.857775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.857678 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.857775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.857715 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8h82\" (UniqueName: \"kubernetes.io/projected/d7b8e021-a67e-4d55-b4a7-b5863c65a981-kube-api-access-s8h82\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.857775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:36.857726 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b8e021-a67e-4d55-b4a7-b5863c65a981-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:37.220999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.211894 5607 generic.go:356] "Generic (PLEG): container finished" podID="d7b8e021-a67e-4d55-b4a7-b5863c65a981" containerID="1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959" exitCode=0 Oct 30 22:35:37.220999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.212098 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" event={"ID":"d7b8e021-a67e-4d55-b4a7-b5863c65a981","Type":"ContainerDied","Data":"1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959"} Oct 30 22:35:37.220999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.212146 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" event={"ID":"d7b8e021-a67e-4d55-b4a7-b5863c65a981","Type":"ContainerDied","Data":"732211d528e195b4ced6c597fb37aa71487089c1fe2e72a6b645808324698e57"} Oct 30 22:35:37.220999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.212170 5607 scope.go:122] "RemoveContainer" containerID="1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959" Oct 30 22:35:37.220999 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.212298 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865" Oct 30 22:35:37.226651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.226628 5607 scope.go:122] "RemoveContainer" containerID="1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959" Oct 30 22:35:37.226990 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:35:37.226944 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959\": container with ID starting with 1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959 not found: ID does not exist" containerID="1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959" Oct 30 22:35:37.227130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.227000 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959"} err="failed to get container status \"1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959\": rpc error: code = NotFound desc = could not find container \"1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959\": container with ID starting with 1d41e1f2e717fa1ca91ec7dc85a77164107f1c56e62fcbfb1499d51c1162d959 not found: ID does not exist" Oct 30 22:35:37.383221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.383193 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865"] Oct 30 22:35:37.385836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:37.385814 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1875b4-f5f9dd55-xr865"] Oct 30 22:35:38.871728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:35:38.871694 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b8e021-a67e-4d55-b4a7-b5863c65a981" path="/var/lib/kubelet/pods/d7b8e021-a67e-4d55-b4a7-b5863c65a981/volumes" Oct 30 22:35:52.865062 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:35:52.865021 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:36:00.903924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:00.903891 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn"] Oct 30 22:36:00.904753 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:00.904730 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d7b8e021-a67e-4d55-b4a7-b5863c65a981" containerName="maas-api" Oct 30 22:36:00.904753 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:00.904753 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b8e021-a67e-4d55-b4a7-b5863c65a981" containerName="maas-api" Oct 30 22:36:00.904975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:00.904943 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b8e021-a67e-4d55-b4a7-b5863c65a981" containerName="maas-api" Oct 30 22:36:00.908388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:00.908366 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:00.962349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:00.962321 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-74bb81-serving-cert\"" Oct 30 22:36:01.016487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.016450 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn"] Oct 30 22:36:01.029059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.029027 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.029224 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.029092 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.029305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.029278 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xv5\" (UniqueName: \"kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.029388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.029370 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.130264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.130226 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xv5\" (UniqueName: \"kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.130457 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.130315 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.130457 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:01.130448 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-serving-cert: secret "maas-api-e2e-shared-b-w0-74bb81-serving-cert" not found Oct 30 22:36:01.130588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.130508 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.130588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.130556 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.130707 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:01.130602 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls podName:0721b687-47df-4042-b54f-50f781677f0d nodeName:}" failed. No retries permitted until 2026-10-30 22:36:01.630578273 +0000 UTC m=+2941.031134373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls") pod "maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" (UID: "0721b687-47df-4042-b54f-50f781677f0d") : secret "maas-api-e2e-shared-b-w0-74bb81-serving-cert" not found Oct 30 22:36:01.131108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.131088 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.133084 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.133062 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.196226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.196204 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xv5\" (UniqueName: \"kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.638445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:01.638358 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:01.638605 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:01.638487 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-serving-cert: secret "maas-api-e2e-shared-b-w0-74bb81-serving-cert" not found Oct 30 22:36:01.638605 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:01.638551 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls podName:0721b687-47df-4042-b54f-50f781677f0d nodeName:}" failed. No retries permitted until 2026-10-30 22:36:02.638538148 +0000 UTC m=+2942.039094247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls") pod "maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" (UID: "0721b687-47df-4042-b54f-50f781677f0d") : secret "maas-api-e2e-shared-b-w0-74bb81-serving-cert" not found Oct 30 22:36:02.650135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:02.650105 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:02.652607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:02.652579 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:02.722439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:02.722371 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:03.099285 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:36:03.099241 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0721b687_47df_4042_b54f_50f781677f0d.slice/crio-e2cd5cf488e26dd559de91fadfadb4c10f4f1ba3fd8a73d139848180af94c7cb WatchSource:0}: Error finding container e2cd5cf488e26dd559de91fadfadb4c10f4f1ba3fd8a73d139848180af94c7cb: Status 404 returned error can't find the container with id e2cd5cf488e26dd559de91fadfadb4c10f4f1ba3fd8a73d139848180af94c7cb Oct 30 22:36:03.102562 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:03.102540 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn"] Oct 30 22:36:03.383221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:03.383152 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" event={"ID":"0721b687-47df-4042-b54f-50f781677f0d","Type":"ContainerStarted","Data":"e2cd5cf488e26dd559de91fadfadb4c10f4f1ba3fd8a73d139848180af94c7cb"} Oct 30 22:36:04.391537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:04.391456 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" event={"ID":"0721b687-47df-4042-b54f-50f781677f0d","Type":"ContainerStarted","Data":"c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5"} Oct 30 22:36:04.391914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:04.391569 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:04.513529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:04.513482 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" podStartSLOduration=3.561982489 podStartE2EDuration="4.513469141s" totalImagesPullingTime="951.486652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:04.511270003 +0000 UTC m=+2943.911826121" watchObservedRunningTime="2026-10-30 22:36:04.513469141 +0000 UTC m=+2943.914025261" Oct 30 22:36:10.403764 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:10.403732 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:36:20.626718 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:20.626678 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z"] Oct 30 22:36:20.627229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:20.626892 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" containerID="cri-o://0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc" gracePeriod=30 Oct 30 22:36:22.863941 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:22.863908 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:36:25.787326 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:25.787286 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 22:36:25.787841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:25.787364 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 22:36:25.897459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:25.897408 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:36:26.036761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.036734 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.036784 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.036985 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037011 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.037258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037085 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037108 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037149 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037294 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.037512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037449 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.037512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037204 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037530 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.037633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037537 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:26.037633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.037575 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket\") pod \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\" (UID: \"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d\") " Oct 30 22:36:26.038404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.038134 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.038651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.038628 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.038813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.038797 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.038915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.038903 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.039052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.039037 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.039156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.039143 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.043515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.042267 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:26.043515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.042324 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:26.043515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.042464 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "kube-api-access-xdff7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:26.046325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.044056 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy" pod "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" (UID: "85501e6a-bd2f-4f0d-9f37-9fa6ee18950d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:26.140944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.140915 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdff7\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-kube-api-access-xdff7\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.140944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.140944 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.141175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.140983 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.141175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.140997 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:26.540563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.540523 5607 generic.go:356] "Generic (PLEG): container finished" podID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerID="0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc" exitCode=0 Oct 30 22:36:26.540778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.540577 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" Oct 30 22:36:26.540778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.540605 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" event={"ID":"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d","Type":"ContainerDied","Data":"0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc"} Oct 30 22:36:26.540778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.540660 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z" event={"ID":"85501e6a-bd2f-4f0d-9f37-9fa6ee18950d","Type":"ContainerDied","Data":"789c28f0f604021531422c605903d3b03292d05982b83f7c2a855f529ad64b9e"} Oct 30 22:36:26.540778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.540682 5607 scope.go:122] "RemoveContainer" containerID="0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc" Oct 30 22:36:26.552518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.552499 5607 scope.go:122] "RemoveContainer" containerID="0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc" Oct 30 22:36:26.552777 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:26.552756 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc\": container with ID starting with 0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc not found: ID does not exist" containerID="0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc" Oct 30 22:36:26.552818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.552785 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc"} err="failed to get container status \"0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc\": rpc error: code = NotFound desc = could not find container \"0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc\": container with ID starting with 0d9695d68e275ce59771c62459b8345c8483cc4a5ce6a7fe0413362e2028abfc not found: ID does not exist" Oct 30 22:36:26.767883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.767854 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z"] Oct 30 22:36:26.772103 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.772082 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-4wc9z"] Oct 30 22:36:26.871256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:26.871199 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" path="/var/lib/kubelet/pods/85501e6a-bd2f-4f0d-9f37-9fa6ee18950d/volumes" Oct 30 22:36:31.813984 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:31.813919 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b"] Oct 30 22:36:31.814510 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:31.814155 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" containerID="cri-o://c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53" gracePeriod=30 Oct 30 22:36:31.865059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:31.865029 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4"] Oct 30 22:36:31.865209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:31.865187 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" containerID="cri-o://f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818" gracePeriod=30 Oct 30 22:36:36.756114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:36.756084 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:36.756487 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:36.756131 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:37.146521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.146489 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:36:37.177162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.177142 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:36:37.258979 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.258931 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.258985 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259018 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259072 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259123 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259175 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259201 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259242 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259334 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259351 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.259512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259414 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259447 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.259512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259480 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259529 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259560 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259578 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.259807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259601 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.259807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259741 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.260304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259890 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket\") pod \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\" (UID: \"af51ad9f-7f7a-4b81-9fcc-6b507843e45a\") " Oct 30 22:36:37.260304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259922 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z\") pod \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\" (UID: \"69672d21-b1d9-49ed-bf6b-a92596d2dd04\") " Oct 30 22:36:37.260304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.259968 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.260304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.260163 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.260572 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.260547 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.260649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.260579 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.260649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.260595 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.260649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.260609 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.260861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.260834 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.261312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.261249 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.261312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.261263 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:37.261530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.261430 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:37.261530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.261466 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.261671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.261529 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.262181 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.262149 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.262385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.262350 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:37.262474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.262440 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:37.262534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.262478 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "kube-api-access-vjmb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:37.262599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.262577 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z" pod "69672d21-b1d9-49ed-bf6b-a92596d2dd04" (UID: "69672d21-b1d9-49ed-bf6b-a92596d2dd04"). InnerVolumeSpecName "kube-api-access-t9s5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:37.262635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.262619 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:37.263388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.263369 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:37.263454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.263405 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token" pod "af51ad9f-7f7a-4b81-9fcc-6b507843e45a" (UID: "af51ad9f-7f7a-4b81-9fcc-6b507843e45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:37.362070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362006 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362030 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362039 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmb5\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-kube-api-access-vjmb5\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362050 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362058 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362066 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362075 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362085 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362093 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362101 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362109 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362117 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69672d21-b1d9-49ed-bf6b-a92596d2dd04-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362125 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af51ad9f-7f7a-4b81-9fcc-6b507843e45a-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.362363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.362133 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9s5z\" (UniqueName: \"kubernetes.io/projected/69672d21-b1d9-49ed-bf6b-a92596d2dd04-kube-api-access-t9s5z\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:37.621915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.621831 5607 generic.go:356] "Generic (PLEG): container finished" podID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerID="f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818" exitCode=0 Oct 30 22:36:37.621915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.621888 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" Oct 30 22:36:37.621915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.621902 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" event={"ID":"af51ad9f-7f7a-4b81-9fcc-6b507843e45a","Type":"ContainerDied","Data":"f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818"} Oct 30 22:36:37.622220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.622011 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4" event={"ID":"af51ad9f-7f7a-4b81-9fcc-6b507843e45a","Type":"ContainerDied","Data":"0840d9f060b5b30d99a61a53519b76a223adc7e8187e00bbd338a290d1719fe3"} Oct 30 22:36:37.622220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.622037 5607 scope.go:122] "RemoveContainer" containerID="f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818" Oct 30 22:36:37.624031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.623841 5607 generic.go:356] "Generic (PLEG): container finished" podID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerID="c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53" exitCode=0 Oct 30 22:36:37.624031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.623994 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" Oct 30 22:36:37.624231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.624041 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" event={"ID":"69672d21-b1d9-49ed-bf6b-a92596d2dd04","Type":"ContainerDied","Data":"c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53"} Oct 30 22:36:37.624231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.624093 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b" event={"ID":"69672d21-b1d9-49ed-bf6b-a92596d2dd04","Type":"ContainerDied","Data":"7038e0298513674112d7e403712852433387e5664e0a17f83f085d632ce0f38b"} Oct 30 22:36:37.636548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.636529 5607 scope.go:122] "RemoveContainer" containerID="f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818" Oct 30 22:36:37.636873 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:37.636851 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818\": container with ID starting with f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818 not found: ID does not exist" containerID="f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818" Oct 30 22:36:37.636942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.636882 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818"} err="failed to get container status \"f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818\": rpc error: code = NotFound desc = could not find container \"f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818\": container with ID starting with f95941a14a8b5c13e613f4fd5816550b575b23215a591b152da42d9dd211f818 not found: ID does not exist" Oct 30 22:36:37.636942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.636904 5607 scope.go:122] "RemoveContainer" containerID="c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53" Oct 30 22:36:37.647053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.647035 5607 scope.go:122] "RemoveContainer" containerID="c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53" Oct 30 22:36:37.647293 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:37.647276 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53\": container with ID starting with c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53 not found: ID does not exist" containerID="c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53" Oct 30 22:36:37.647353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.647299 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53"} err="failed to get container status \"c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53\": rpc error: code = NotFound desc = could not find container \"c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53\": container with ID starting with c6239ed7352604979647686783813844e6e8e3f4f75c0661d8c60f7ef7d27b53 not found: ID does not exist" Oct 30 22:36:37.793439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.793414 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4"] Oct 30 22:36:37.799563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.799539 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-l7px4"] Oct 30 22:36:37.912440 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.912380 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b"] Oct 30 22:36:37.920105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:37.920079 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4vz5b"] Oct 30 22:36:38.872353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:38.872322 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" path="/var/lib/kubelet/pods/69672d21-b1d9-49ed-bf6b-a92596d2dd04/volumes" Oct 30 22:36:38.872765 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:36:38.872752 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" path="/var/lib/kubelet/pods/af51ad9f-7f7a-4b81-9fcc-6b507843e45a/volumes" Oct 30 22:36:52.863767 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:36:52.863731 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:37:01.313478 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:37:01.313446 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:37:01.316215 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:37:01.316196 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:37:22.863665 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:37:22.863625 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:37:52.863933 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:37:52.863897 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:38:06.428549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:06.428511 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb"] Oct 30 22:38:06.429088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:06.428725 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" containerID="cri-o://ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8" gracePeriod=30 Oct 30 22:38:10.962255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:10.962171 5607 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:38:10.962255 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:10.962237 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:38:11.705343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.705284 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:38:11.775654 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775628 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.775813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775675 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.775813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775704 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.775813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775736 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.775813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775803 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.776065 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775840 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.776065 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775880 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.776188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776062 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.776188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776066 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:11.776188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.775944 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:11.776188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776096 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token\") pod \"9120d792-d37d-4255-82df-8cf3be1fcff2\" (UID: \"9120d792-d37d-4255-82df-8cf3be1fcff2\") " Oct 30 22:38:11.776409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776256 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:11.776409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776279 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:11.776409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776180 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:11.776816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776792 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.776882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776824 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.776882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776840 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.776882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776855 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9120d792-d37d-4255-82df-8cf3be1fcff2-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.776882 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.776871 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.778234 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.778211 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:38:11.778336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.778274 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:11.778336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.778285 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "kube-api-access-ktpnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:11.778336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.778310 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token" pod "9120d792-d37d-4255-82df-8cf3be1fcff2" (UID: "9120d792-d37d-4255-82df-8cf3be1fcff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:11.877509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.877480 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.877509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.877505 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.877509 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.877515 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktpnb\" (UniqueName: \"kubernetes.io/projected/9120d792-d37d-4255-82df-8cf3be1fcff2-kube-api-access-ktpnb\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:11.877708 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:11.877525 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9120d792-d37d-4255-82df-8cf3be1fcff2-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:12.226638 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.226606 5607 generic.go:356] "Generic (PLEG): container finished" podID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerID="ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8" exitCode=0 Oct 30 22:38:12.227088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.226654 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" Oct 30 22:38:12.227088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.226690 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" event={"ID":"9120d792-d37d-4255-82df-8cf3be1fcff2","Type":"ContainerDied","Data":"ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8"} Oct 30 22:38:12.227088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.226747 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb" event={"ID":"9120d792-d37d-4255-82df-8cf3be1fcff2","Type":"ContainerDied","Data":"e74f3166bf8225c6864408c0891aba8260b98b3566568a98ac89760379475ebe"} Oct 30 22:38:12.227088 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.226776 5607 scope.go:122] "RemoveContainer" containerID="ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8" Oct 30 22:38:12.237860 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.237842 5607 scope.go:122] "RemoveContainer" containerID="ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8" Oct 30 22:38:12.238141 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:38:12.238122 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8\": container with ID starting with ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8 not found: ID does not exist" containerID="ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8" Oct 30 22:38:12.238206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.238151 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8"} err="failed to get container status \"ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8\": rpc error: code = NotFound desc = could not find container \"ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8\": container with ID starting with ee57330db6489cbe44a8be2ef1464e8d8f9004635fb5b94a2943b048486c43b8 not found: ID does not exist" Oct 30 22:38:12.398459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.398431 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb"] Oct 30 22:38:12.449534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.449510 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-m9ztb"] Oct 30 22:38:12.879430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:38:12.879397 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" path="/var/lib/kubelet/pods/9120d792-d37d-4255-82df-8cf3be1fcff2/volumes" Oct 30 22:38:22.863454 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:38:22.863415 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:38:52.866023 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:38:52.865982 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:39:22.863990 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:39:22.863933 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:39:52.864983 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:39:52.864922 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:40:22.863483 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:40:22.863443 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:40:52.864508 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:40:52.864467 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:41:13.492320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492232 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6"] Oct 30 22:41:13.492884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492870 5607 cpu_manager.go:414] "containerMap: removing container" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" Oct 30 22:41:13.492929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492885 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" Oct 30 22:41:13.492929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492904 5607 cpu_manager.go:414] "containerMap: removing container" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" Oct 30 22:41:13.492929 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492910 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" Oct 30 22:41:13.493089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492939 5607 cpu_manager.go:414] "containerMap: removing container" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" Oct 30 22:41:13.493089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492944 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" Oct 30 22:41:13.493089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492982 5607 cpu_manager.go:414] "containerMap: removing container" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" Oct 30 22:41:13.493089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.492988 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" Oct 30 22:41:13.493089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.493088 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="af51ad9f-7f7a-4b81-9fcc-6b507843e45a" containerName="istio-proxy" Oct 30 22:41:13.493260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.493104 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="85501e6a-bd2f-4f0d-9f37-9fa6ee18950d" containerName="istio-proxy" Oct 30 22:41:13.493260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.493112 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="9120d792-d37d-4255-82df-8cf3be1fcff2" containerName="istio-proxy" Oct 30 22:41:13.493260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.493119 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="69672d21-b1d9-49ed-bf6b-a92596d2dd04" containerName="istio-proxy" Oct 30 22:41:13.496530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.496514 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.598447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.598419 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.598599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.598455 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46ft\" (UniqueName: \"kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft\") pod \"payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.601830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.601806 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-421286-dockercfg-55pf4\"" Oct 30 22:41:13.601830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.601830 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-421286\"" Oct 30 22:41:13.605535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.605512 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6"] Oct 30 22:41:13.699271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.699240 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.699271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.699281 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46ft\" (UniqueName: \"kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft\") pod \"payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.700264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.700242 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.762120 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.762059 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46ft\" (UniqueName: \"kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft\") pod \"payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:13.810666 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:13.810640 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:14.189802 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:41:14.189768 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c486f28_0c4c_4065_910a_3d590c82003d.slice/crio-a44efa2f2317ec4933dcc50236b8d2b8c81593bb636ee01292552f2af35864d3 WatchSource:0}: Error finding container a44efa2f2317ec4933dcc50236b8d2b8c81593bb636ee01292552f2af35864d3: Status 404 returned error can't find the container with id a44efa2f2317ec4933dcc50236b8d2b8c81593bb636ee01292552f2af35864d3 Oct 30 22:41:14.191428 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:14.191405 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6"] Oct 30 22:41:14.415033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:14.414996 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" event={"ID":"3c486f28-0c4c-4065-910a-3d590c82003d","Type":"ContainerStarted","Data":"30bf3c9968640ed90e54e169e5f1c4c7ff4c2a4547ce3bd21b22934cb89361f3"} Oct 30 22:41:14.415366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:14.415060 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" event={"ID":"3c486f28-0c4c-4065-910a-3d590c82003d","Type":"ContainerStarted","Data":"a44efa2f2317ec4933dcc50236b8d2b8c81593bb636ee01292552f2af35864d3"} Oct 30 22:41:14.415366 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:14.415100 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:14.534941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:14.534851 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" podStartSLOduration=1.534838019 podStartE2EDuration="1.534838019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:14.533659601 +0000 UTC m=+3253.934215722" watchObservedRunningTime="2026-10-30 22:41:14.534838019 +0000 UTC m=+3253.935394139" Oct 30 22:41:22.865321 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:41:22.865285 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:41:25.424574 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:25.424543 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:41:34.118242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:34.118201 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l"] Oct 30 22:41:34.118692 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:34.118402 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" containerID="cri-o://7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad" gracePeriod=30 Oct 30 22:41:37.608236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:37.608172 5607 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:37.608856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:37.608237 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:39.389855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.389804 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:41:39.475179 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475146 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475212 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475317 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475250 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475324 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475384 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475475 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475512 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475525 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:39.475772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475591 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475763 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket\") pod \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\" (UID: \"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0\") " Oct 30 22:41:39.475887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475769 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:39.475887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475787 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:39.475887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.475787 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:39.476097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.476073 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:39.476594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.476464 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.476594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.476494 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.476594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.476511 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.476594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.476525 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.476594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.476542 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.477871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.477845 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "kube-api-access-2g2rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:39.478006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.477911 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:39.478064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.478002 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:39.478283 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.478251 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token" pod "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" (UID: "c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:39.578241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.578203 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2rp\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-kube-api-access-2g2rp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.578241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.578243 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.578517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.578262 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.578517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.578274 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:39.579339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.579307 5607 generic.go:356] "Generic (PLEG): container finished" podID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerID="7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad" exitCode=0 Oct 30 22:41:39.579485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.579368 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" Oct 30 22:41:39.579485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.579392 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" event={"ID":"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0","Type":"ContainerDied","Data":"7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad"} Oct 30 22:41:39.579485 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.579462 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l" event={"ID":"c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0","Type":"ContainerDied","Data":"c9c2a0d0dd573f88760ea977dfaeddd14400c1d95e5cb33a815d16ef12d97c84"} Oct 30 22:41:39.579688 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.579491 5607 scope.go:122] "RemoveContainer" containerID="7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad" Oct 30 22:41:39.593350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.593324 5607 scope.go:122] "RemoveContainer" containerID="7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad" Oct 30 22:41:39.593686 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:41:39.593652 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad\": container with ID starting with 7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad not found: ID does not exist" containerID="7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad" Oct 30 22:41:39.593686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.593678 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad"} err="failed to get container status \"7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad\": rpc error: code = NotFound desc = could not find container \"7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad\": container with ID starting with 7832b8388deb6c071514373b0381571e6a315844db44f25f50a26ee633f690ad not found: ID does not exist" Oct 30 22:41:39.749034 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.749010 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l"] Oct 30 22:41:39.754600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:39.754573 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-4lh6l"] Oct 30 22:41:40.870865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:41:40.870835 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" path="/var/lib/kubelet/pods/c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0/volumes" Oct 30 22:41:52.864895 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:41:52.864855 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:42:01.356251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:01.356227 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:42:01.360218 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:01.360200 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:42:21.083153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.083119 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx"] Oct 30 22:42:21.083969 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.083930 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" Oct 30 22:42:21.084089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.083972 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" Oct 30 22:42:21.084204 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.084189 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c37f3f-7ae1-441a-80f6-a7e5e245dfc0" containerName="istio-proxy" Oct 30 22:42:21.088864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.088842 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.117520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.117491 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.117626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.117526 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sns7w\" (UniqueName: \"kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.117726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.117714 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.117772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.117745 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.196011 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.195968 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-421286-serving-cert\"" Oct 30 22:42:21.198901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.198877 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx"] Oct 30 22:42:21.218598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.218563 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.218761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.218614 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.218761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.218671 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.218761 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.218699 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sns7w\" (UniqueName: \"kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.218978 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:42:21.218853 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-serving-cert: secret "maas-api-e2e-extmod-a-421286-serving-cert" not found Oct 30 22:42:21.219044 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:42:21.219005 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls podName:eec8660c-601d-4bdc-a880-4e6d9e2c1415 nodeName:}" failed. No retries permitted until 2026-10-30 22:42:21.718981654 +0000 UTC m=+3321.119537776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls") pod "maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" (UID: "eec8660c-601d-4bdc-a880-4e6d9e2c1415") : secret "maas-api-e2e-extmod-a-421286-serving-cert" not found Oct 30 22:42:21.219590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.219463 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.221388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.221369 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.313789 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.313754 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sns7w\" (UniqueName: \"kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.733520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.733486 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:21.736117 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:21.736092 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\") pod \"maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:22.003274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:22.003189 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:22.395557 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:42:22.395522 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec8660c_601d_4bdc_a880_4e6d9e2c1415.slice/crio-3b3fa9c85d05ed1d667d36c61a729cdea4a24cadb6875f20743f28ed41912ea5 WatchSource:0}: Error finding container 3b3fa9c85d05ed1d667d36c61a729cdea4a24cadb6875f20743f28ed41912ea5: Status 404 returned error can't find the container with id 3b3fa9c85d05ed1d667d36c61a729cdea4a24cadb6875f20743f28ed41912ea5 Oct 30 22:42:22.397258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:22.397237 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:42:22.398143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:22.398117 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx"] Oct 30 22:42:22.865297 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:42:22.865251 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:42:22.873627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:22.873586 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" event={"ID":"eec8660c-601d-4bdc-a880-4e6d9e2c1415","Type":"ContainerStarted","Data":"3b3fa9c85d05ed1d667d36c61a729cdea4a24cadb6875f20743f28ed41912ea5"} Oct 30 22:42:23.874632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:23.874593 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" event={"ID":"eec8660c-601d-4bdc-a880-4e6d9e2c1415","Type":"ContainerStarted","Data":"30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc"} Oct 30 22:42:23.875101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:23.874768 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:24.001223 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:24.001165 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" podStartSLOduration=1.9881466909999999 podStartE2EDuration="3.001149612s" totalImagesPullingTime="1.013002921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:23.998792238 +0000 UTC m=+3323.399348360" watchObservedRunningTime="2026-10-30 22:42:24.001149612 +0000 UTC m=+3323.401705732" Oct 30 22:42:29.886527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:29.886495 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:42:50.740435 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.740346 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85"] Oct 30 22:42:50.744914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.744890 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.766893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.766867 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlspr\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.766909 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.766941 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.767069 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.767106 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.767222 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.767259 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.767371 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.767453 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.767421 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868041 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868012 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868088 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868137 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868212 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlspr\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868240 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868269 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868303 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868339 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.868652 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.868427 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.870877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.869138 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.870877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.869203 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.870877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.869458 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.870877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.869702 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.870877 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.870050 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.871396 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.871370 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.872144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.872115 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.894163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.894144 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-421286-openshift-default-dockercfg-qlszb\"" Oct 30 22:42:50.931886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.931864 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlspr\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.932038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.932023 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:50.956345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:50.956318 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85"] Oct 30 22:42:51.058563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:51.058496 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:51.485208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:51.485185 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85"] Oct 30 22:42:51.485394 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:51.485363 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:42:51.485443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:51.485426 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:42:51.485482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:51.485458 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:42:52.056815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:52.056778 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" event={"ID":"51e967b0-69dd-46e9-99e5-796a073a73d8","Type":"ContainerStarted","Data":"47ebefc2061faeb33b34b74f8357a0f2b5e1b1fdd6860577b1377f89f84a1ac0"} Oct 30 22:42:52.057281 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:52.056832 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" event={"ID":"51e967b0-69dd-46e9-99e5-796a073a73d8","Type":"ContainerStarted","Data":"a3a4ff0dbe8e16a2f40a4b2b90ecc37d6e565432f7716058246869fe14fb4171"} Oct 30 22:42:52.059158 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:52.059139 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:52.060857 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:52.060839 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 22:42:52.060920 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:52.060889 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 22:42:52.189983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:52.184991 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podStartSLOduration=2.184970471 podStartE2EDuration="2.184970471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:52.180824748 +0000 UTC m=+3351.581380870" watchObservedRunningTime="2026-10-30 22:42:52.184970471 +0000 UTC m=+3351.585526586" Oct 30 22:42:52.863768 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:42:52.863732 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:42:53.059097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:53.059061 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 22:42:53.059535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:53.059122 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 22:42:54.059521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:54.059484 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 22:42:54.059914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:54.059542 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 22:42:55.059324 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:55.059290 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 22:42:55.059615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:55.059365 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 22:42:56.059617 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:56.059586 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 22:42:56.060055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:56.059635 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 22:42:57.059003 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:57.058966 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 22:42:57.059227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:57.059042 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 22:42:58.064407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:58.064378 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:58.064881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:58.064858 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:42:58.065464 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:42:58.065442 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:43:22.864213 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:43:22.864174 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:43:34.824125 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.824087 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq"] Oct 30 22:43:34.828300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.828284 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:34.887081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.887057 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq"] Oct 30 22:43:34.935235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.935177 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9a60dd-dockercfg-zlt5q\"" Oct 30 22:43:34.935597 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.935552 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9a60dd\"" Oct 30 22:43:34.974680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.974643 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:34.974680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.974697 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmf44\" (UniqueName: \"kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44\") pod \"payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:34.989190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.989164 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p"] Oct 30 22:43:34.993429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:34.993410 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.028476 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.028448 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p"] Oct 30 22:43:35.076401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.076346 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:35.076401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.076377 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmf44\" (UniqueName: \"kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44\") pod \"payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:35.076543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.076446 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptdm\" (UniqueName: \"kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm\") pod \"payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.076543 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.076495 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.077001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.076982 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:35.138713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.138688 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmf44\" (UniqueName: \"kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44\") pod \"payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:35.142632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.142617 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:35.178210 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.178182 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptdm\" (UniqueName: \"kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm\") pod \"payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.178320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.178242 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.179155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.179131 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.243914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.243833 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptdm\" (UniqueName: \"kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm\") pod \"payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.307841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.307815 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:35.523304 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:43:35.523271 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805fd112_9e46_4ac0_b482_5aab9c6ec1b8.slice/crio-62a015b03fb54e626ce7384c1f6985c691d020c9fec58bf201e6eedd780289b5 WatchSource:0}: Error finding container 62a015b03fb54e626ce7384c1f6985c691d020c9fec58bf201e6eedd780289b5: Status 404 returned error can't find the container with id 62a015b03fb54e626ce7384c1f6985c691d020c9fec58bf201e6eedd780289b5 Oct 30 22:43:35.526219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.526195 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq"] Oct 30 22:43:35.682432 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:43:35.681699 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bf32a3_efed_4c9a_bbd1_64815f1b60ec.slice/crio-cdd8759fb662b1df1fd550cae0d829dcb308a78a025dea7370c323f82d7c4aeb WatchSource:0}: Error finding container cdd8759fb662b1df1fd550cae0d829dcb308a78a025dea7370c323f82d7c4aeb: Status 404 returned error can't find the container with id cdd8759fb662b1df1fd550cae0d829dcb308a78a025dea7370c323f82d7c4aeb Oct 30 22:43:35.684578 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:35.684552 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p"] Oct 30 22:43:36.108149 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.108112 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2"] Oct 30 22:43:36.112589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.112565 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.197698 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.197667 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.197901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.197724 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lwx\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.197898 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.198001 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.198021 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.198041 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.198059 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.198089 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.198216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.198122 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.299632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299594 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.299842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299657 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lwx\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.299842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299746 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.299842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299817 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.299842 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299838 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299866 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299921 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.299992 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.300044 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.300098 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.300225 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.300282 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.300347 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.300535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.300480 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.302454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.302432 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.302607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.302591 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.320558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.320530 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2"] Oct 30 22:43:36.348703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.348666 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" event={"ID":"48bf32a3-efed-4c9a-bbd1-64815f1b60ec","Type":"ContainerStarted","Data":"352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522"} Oct 30 22:43:36.349058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.349032 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:36.349168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.349080 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" event={"ID":"48bf32a3-efed-4c9a-bbd1-64815f1b60ec","Type":"ContainerStarted","Data":"cdd8759fb662b1df1fd550cae0d829dcb308a78a025dea7370c323f82d7c4aeb"} Oct 30 22:43:36.350549 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.350517 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" event={"ID":"805fd112-9e46-4ac0-b482-5aab9c6ec1b8","Type":"ContainerStarted","Data":"fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2"} Oct 30 22:43:36.350701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.350570 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" event={"ID":"805fd112-9e46-4ac0-b482-5aab9c6ec1b8","Type":"ContainerStarted","Data":"62a015b03fb54e626ce7384c1f6985c691d020c9fec58bf201e6eedd780289b5"} Oct 30 22:43:36.350701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.350596 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:36.362069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.362007 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.362250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.362230 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lwx\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.427159 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.427130 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:36.467458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.467409 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" podStartSLOduration=2.467396883 podStartE2EDuration="2.467396883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:36.467109851 +0000 UTC m=+3395.867665972" watchObservedRunningTime="2026-10-30 22:43:36.467396883 +0000 UTC m=+3395.867953004" Oct 30 22:43:36.588133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.588082 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" podStartSLOduration=2.588062668 podStartE2EDuration="2.588062668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:36.586180393 +0000 UTC m=+3395.986736528" watchObservedRunningTime="2026-10-30 22:43:36.588062668 +0000 UTC m=+3395.988618791" Oct 30 22:43:36.851794 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:43:36.851760 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a5bbc0_5baf_43db_a70a_ea81a102c270.slice/crio-c2a1195f2b7d27ccb3720cd65f8af4b5d3453014f6996f0f6a58d6bfa10e8aff WatchSource:0}: Error finding container c2a1195f2b7d27ccb3720cd65f8af4b5d3453014f6996f0f6a58d6bfa10e8aff: Status 404 returned error can't find the container with id c2a1195f2b7d27ccb3720cd65f8af4b5d3453014f6996f0f6a58d6bfa10e8aff Oct 30 22:43:36.854557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.854521 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:43:36.854637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.854591 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:43:36.854637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.854624 5607 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:43:36.854938 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:36.854919 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2"] Oct 30 22:43:37.358634 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:37.358597 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" event={"ID":"f2a5bbc0-5baf-43db-a70a-ea81a102c270","Type":"ContainerStarted","Data":"1c184760f74456f6f8554e56a918e1d93d6073a118dd873fb74454e654dd90ad"} Oct 30 22:43:37.359159 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:37.358649 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" event={"ID":"f2a5bbc0-5baf-43db-a70a-ea81a102c270","Type":"ContainerStarted","Data":"c2a1195f2b7d27ccb3720cd65f8af4b5d3453014f6996f0f6a58d6bfa10e8aff"} Oct 30 22:43:37.432494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:37.428416 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:37.433423 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:37.433359 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 22:43:37.434483 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:37.433430 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 22:43:37.485897 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:37.485406 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podStartSLOduration=1.485387022 podStartE2EDuration="1.485387022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:37.48193291 +0000 UTC m=+3396.882489030" watchObservedRunningTime="2026-10-30 22:43:37.485387022 +0000 UTC m=+3396.885943144" Oct 30 22:43:38.428565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:38.428529 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 22:43:38.428993 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:38.428585 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 22:43:39.427501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:39.427465 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 22:43:39.427697 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:39.427516 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 22:43:40.428219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:40.428185 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 22:43:40.428637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:40.428236 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 22:43:41.428104 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:41.428068 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 22:43:41.428325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:41.428129 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 22:43:42.429590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:42.429297 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 22:43:42.429590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:42.429357 5607 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 22:43:43.537508 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:43.537456 5607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:43.538052 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:43.538019 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:43.539253 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:43.539231 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:43:47.361751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:47.361718 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:43:47.362159 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:43:47.361797 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:43:52.864854 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:43:52.864822 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:44:22.866365 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:44:22.866316 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:44:31.510663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.510628 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq"] Oct 30 22:44:31.516859 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.516835 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.573167 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.573137 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq"] Oct 30 22:44:31.609113 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.609082 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtfh\" (UniqueName: \"kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh\") pod \"payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.609275 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.609189 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.620170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.620149 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-dcde03\"" Oct 30 22:44:31.620170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.620160 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-dcde03-dockercfg-gds6b\"" Oct 30 22:44:31.710293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.710261 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.710466 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.710449 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtfh\" (UniqueName: \"kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh\") pod \"payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.710909 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.710892 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.777564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.777511 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtfh\" (UniqueName: \"kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh\") pod \"payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:31.833634 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:31.833610 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:32.212152 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:44:32.212118 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5ca4b4_ddf1_4f24_ae00_4b3ee9bea5fc.slice/crio-b3779044d52264daab0150e4a5e65a0019ee37db2ed2209ac23c42c43909a6de WatchSource:0}: Error finding container b3779044d52264daab0150e4a5e65a0019ee37db2ed2209ac23c42c43909a6de: Status 404 returned error can't find the container with id b3779044d52264daab0150e4a5e65a0019ee37db2ed2209ac23c42c43909a6de Oct 30 22:44:32.214967 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:32.214927 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq"] Oct 30 22:44:32.746502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:32.746455 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" event={"ID":"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc","Type":"ContainerStarted","Data":"1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3"} Oct 30 22:44:32.746878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:32.746511 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" event={"ID":"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc","Type":"ContainerStarted","Data":"b3779044d52264daab0150e4a5e65a0019ee37db2ed2209ac23c42c43909a6de"} Oct 30 22:44:32.746878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:32.746542 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:32.871658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:32.871613 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" podStartSLOduration=1.8716002390000002 podStartE2EDuration="1.871600239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:32.870106923 +0000 UTC m=+3452.270663044" watchObservedRunningTime="2026-10-30 22:44:32.871600239 +0000 UTC m=+3452.272156360" Oct 30 22:44:43.756592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:44:43.756564 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:44:52.864550 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:44:52.864513 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:45:00.204190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.204151 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b"] Oct 30 22:45:00.209330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.209305 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.231430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.231408 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb"] Oct 30 22:45:00.235544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.235527 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.237834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.237812 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k"] Oct 30 22:45:00.241792 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.241770 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.254977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.254941 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2"] Oct 30 22:45:00.259197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.259181 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.323894 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.323862 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.324112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.323916 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8twx\" (UniqueName: \"kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx\") pod \"maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.324112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.323969 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.324112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.324065 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qgg\" (UniqueName: \"kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.324112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.324094 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbp2\" (UniqueName: \"kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2\") pod \"maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.324360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.324183 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.324360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.324308 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.324360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.324348 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vnw\" (UniqueName: \"kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.328475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.328449 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b"] Oct 30 22:45:00.334845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.334826 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:45:00.385213 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.385183 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb"] Oct 30 22:45:00.391026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.391003 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k"] Oct 30 22:45:00.393081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.393055 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2"] Oct 30 22:45:00.425062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425032 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qgg\" (UniqueName: \"kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.425257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425075 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbp2\" (UniqueName: \"kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2\") pod \"maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.425257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425159 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.425429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425275 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.425429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425315 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vnw\" (UniqueName: \"kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.425429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425359 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.425429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425395 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8twx\" (UniqueName: \"kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx\") pod \"maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.425665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.425432 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.426168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.426147 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.426917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.426896 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.427365 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.427346 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.428111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.428088 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.488694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.488636 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbp2\" (UniqueName: \"kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2\") pod \"maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.488811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.488788 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qgg\" (UniqueName: \"kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.489527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.489507 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vnw\" (UniqueName: \"kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.490094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.490075 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8twx\" (UniqueName: \"kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx\") pod \"maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.523069 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.523051 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:00.549367 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.549344 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:00.556592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.556567 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:00.575945 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.575915 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:00.909613 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:45:00.909586 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32fe2f2_13fc_4435_870e_c6fdc091f33d.slice/crio-9b8f96e1ccd674cee2fc8e4c7d817959192f2f46be0b95e86287f7e703514695 WatchSource:0}: Error finding container 9b8f96e1ccd674cee2fc8e4c7d817959192f2f46be0b95e86287f7e703514695: Status 404 returned error can't find the container with id 9b8f96e1ccd674cee2fc8e4c7d817959192f2f46be0b95e86287f7e703514695 Oct 30 22:45:00.913191 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.913172 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b"] Oct 30 22:45:00.934015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.933988 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" event={"ID":"a32fe2f2-13fc-4435-870e-c6fdc091f33d","Type":"ContainerStarted","Data":"9b8f96e1ccd674cee2fc8e4c7d817959192f2f46be0b95e86287f7e703514695"} Oct 30 22:45:00.954656 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:45:00.954608 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82d1693_9126_47a7_976f_21bb4a31c856.slice/crio-d4d52520135ea80ba57fe9f4697ba5c9063368513d4effbe078fffc7eb46e806 WatchSource:0}: Error finding container d4d52520135ea80ba57fe9f4697ba5c9063368513d4effbe078fffc7eb46e806: Status 404 returned error can't find the container with id d4d52520135ea80ba57fe9f4697ba5c9063368513d4effbe078fffc7eb46e806 Oct 30 22:45:00.958270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.958247 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb"] Oct 30 22:45:00.959635 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:45:00.959609 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d26170_cead_4921_9e17_37b97153856f.slice/crio-212660efe7ba258c9b47a054e722a4e4cf85dc61591aed39f97db0725c9b0fb8 WatchSource:0}: Error finding container 212660efe7ba258c9b47a054e722a4e4cf85dc61591aed39f97db0725c9b0fb8: Status 404 returned error can't find the container with id 212660efe7ba258c9b47a054e722a4e4cf85dc61591aed39f97db0725c9b0fb8 Oct 30 22:45:00.962339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:00.962317 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k"] Oct 30 22:45:01.016505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.016475 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2"] Oct 30 22:45:01.099417 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:45:01.099388 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a665099_6fc0_4174_8d29_ab64f4eac4b7.slice/crio-caf5ff694b269fa6ddd5f15409acc36c6c0ad1220131c3311601af9d4aa1869d WatchSource:0}: Error finding container caf5ff694b269fa6ddd5f15409acc36c6c0ad1220131c3311601af9d4aa1869d: Status 404 returned error can't find the container with id caf5ff694b269fa6ddd5f15409acc36c6c0ad1220131c3311601af9d4aa1869d Oct 30 22:45:01.944039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.944010 5607 generic.go:356] "Generic (PLEG): container finished" podID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerID="10df07b27c25a4db69b0a5ef31113f33207e55f8c216d5d4d92d60b2d705e4a1" exitCode=60 Oct 30 22:45:01.944445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.944092 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" event={"ID":"a32fe2f2-13fc-4435-870e-c6fdc091f33d","Type":"ContainerDied","Data":"10df07b27c25a4db69b0a5ef31113f33207e55f8c216d5d4d92d60b2d705e4a1"} Oct 30 22:45:01.944445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.944396 5607 scope.go:122] "RemoveContainer" containerID="10df07b27c25a4db69b0a5ef31113f33207e55f8c216d5d4d92d60b2d705e4a1" Oct 30 22:45:01.945855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.945829 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" event={"ID":"b82d1693-9126-47a7-976f-21bb4a31c856","Type":"ContainerStarted","Data":"961094da8d1f38ac6f33ba54269661d15c4b9640b023c7aa07351748cd2235b1"} Oct 30 22:45:01.945997 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.945876 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" event={"ID":"b82d1693-9126-47a7-976f-21bb4a31c856","Type":"ContainerStarted","Data":"d4d52520135ea80ba57fe9f4697ba5c9063368513d4effbe078fffc7eb46e806"} Oct 30 22:45:01.948530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.948448 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" event={"ID":"e8d26170-cead-4921-9e17-37b97153856f","Type":"ContainerStarted","Data":"3f4f78f1bb67f17769bddd360650983cb7f5b93f0bba25841ebef32e42a8c7f2"} Oct 30 22:45:01.948530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.948502 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" event={"ID":"e8d26170-cead-4921-9e17-37b97153856f","Type":"ContainerStarted","Data":"212660efe7ba258c9b47a054e722a4e4cf85dc61591aed39f97db0725c9b0fb8"} Oct 30 22:45:01.950226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.950196 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerStarted","Data":"75520ff58cd6e3c7ef24d2ec59d2f9521f884b0b35091362481730f724b409b4"} Oct 30 22:45:01.950331 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:01.950239 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerStarted","Data":"caf5ff694b269fa6ddd5f15409acc36c6c0ad1220131c3311601af9d4aa1869d"} Oct 30 22:45:02.184471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.184397 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" podStartSLOduration=2.184383274 podStartE2EDuration="2.184383274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.182488886 +0000 UTC m=+3481.583045010" watchObservedRunningTime="2026-10-30 22:45:02.184383274 +0000 UTC m=+3481.584939391" Oct 30 22:45:02.304139 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.304087 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" podStartSLOduration=2.304074254 podStartE2EDuration="2.304074254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.301690833 +0000 UTC m=+3481.702246954" watchObservedRunningTime="2026-10-30 22:45:02.304074254 +0000 UTC m=+3481.704630375" Oct 30 22:45:02.431232 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.431134 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" podStartSLOduration=2.431117255 podStartE2EDuration="2.431117255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.430004901 +0000 UTC m=+3481.830561023" watchObservedRunningTime="2026-10-30 22:45:02.431117255 +0000 UTC m=+3481.831673376" Oct 30 22:45:02.959685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.959656 5607 generic.go:356] "Generic (PLEG): container finished" podID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerID="2ceaee3e91cf9763d46eaa690e72f1d318b16abbbf716b8234416f5808e05f6b" exitCode=60 Oct 30 22:45:02.960152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.959739 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" event={"ID":"a32fe2f2-13fc-4435-870e-c6fdc091f33d","Type":"ContainerDied","Data":"2ceaee3e91cf9763d46eaa690e72f1d318b16abbbf716b8234416f5808e05f6b"} Oct 30 22:45:02.960152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.959806 5607 scope.go:122] "RemoveContainer" containerID="10df07b27c25a4db69b0a5ef31113f33207e55f8c216d5d4d92d60b2d705e4a1" Oct 30 22:45:02.960152 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.960014 5607 scope.go:122] "RemoveContainer" containerID="2ceaee3e91cf9763d46eaa690e72f1d318b16abbbf716b8234416f5808e05f6b" Oct 30 22:45:02.960277 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:02.960249 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b_odh-ai-gateway-infra(a32fe2f2-13fc-4435-870e-c6fdc091f33d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" Oct 30 22:45:02.961591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.961567 5607 generic.go:356] "Generic (PLEG): container finished" podID="b82d1693-9126-47a7-976f-21bb4a31c856" containerID="961094da8d1f38ac6f33ba54269661d15c4b9640b023c7aa07351748cd2235b1" exitCode=60 Oct 30 22:45:02.961709 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.961641 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" event={"ID":"b82d1693-9126-47a7-976f-21bb4a31c856","Type":"ContainerDied","Data":"961094da8d1f38ac6f33ba54269661d15c4b9640b023c7aa07351748cd2235b1"} Oct 30 22:45:02.961861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.961845 5607 scope.go:122] "RemoveContainer" containerID="961094da8d1f38ac6f33ba54269661d15c4b9640b023c7aa07351748cd2235b1" Oct 30 22:45:02.963429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.963410 5607 generic.go:356] "Generic (PLEG): container finished" podID="e8d26170-cead-4921-9e17-37b97153856f" containerID="3f4f78f1bb67f17769bddd360650983cb7f5b93f0bba25841ebef32e42a8c7f2" exitCode=60 Oct 30 22:45:02.963492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.963470 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" event={"ID":"e8d26170-cead-4921-9e17-37b97153856f","Type":"ContainerDied","Data":"3f4f78f1bb67f17769bddd360650983cb7f5b93f0bba25841ebef32e42a8c7f2"} Oct 30 22:45:02.963682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.963663 5607 scope.go:122] "RemoveContainer" containerID="3f4f78f1bb67f17769bddd360650983cb7f5b93f0bba25841ebef32e42a8c7f2" Oct 30 22:45:02.965616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.965491 5607 generic.go:356] "Generic (PLEG): container finished" podID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerID="75520ff58cd6e3c7ef24d2ec59d2f9521f884b0b35091362481730f724b409b4" exitCode=60 Oct 30 22:45:02.965616 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.965523 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerDied","Data":"75520ff58cd6e3c7ef24d2ec59d2f9521f884b0b35091362481730f724b409b4"} Oct 30 22:45:02.965825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:02.965751 5607 scope.go:122] "RemoveContainer" containerID="75520ff58cd6e3c7ef24d2ec59d2f9521f884b0b35091362481730f724b409b4" Oct 30 22:45:03.975033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.974999 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerStarted","Data":"ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab"} Oct 30 22:45:03.977300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.977281 5607 scope.go:122] "RemoveContainer" containerID="2ceaee3e91cf9763d46eaa690e72f1d318b16abbbf716b8234416f5808e05f6b" Oct 30 22:45:03.977565 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:03.977541 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b_odh-ai-gateway-infra(a32fe2f2-13fc-4435-870e-c6fdc091f33d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" Oct 30 22:45:03.978980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.978942 5607 generic.go:356] "Generic (PLEG): container finished" podID="b82d1693-9126-47a7-976f-21bb4a31c856" containerID="28ff42387c5a4b42de098f089bc5940794bf8e340a9431e7e19fa071c87a4c4e" exitCode=60 Oct 30 22:45:03.979098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.978984 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" event={"ID":"b82d1693-9126-47a7-976f-21bb4a31c856","Type":"ContainerDied","Data":"28ff42387c5a4b42de098f089bc5940794bf8e340a9431e7e19fa071c87a4c4e"} Oct 30 22:45:03.979098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.979031 5607 scope.go:122] "RemoveContainer" containerID="961094da8d1f38ac6f33ba54269661d15c4b9640b023c7aa07351748cd2235b1" Oct 30 22:45:03.979269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.979248 5607 scope.go:122] "RemoveContainer" containerID="28ff42387c5a4b42de098f089bc5940794bf8e340a9431e7e19fa071c87a4c4e" Oct 30 22:45:03.979553 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:03.979530 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb_odh-ai-gateway-infra(b82d1693-9126-47a7-976f-21bb4a31c856)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" Oct 30 22:45:03.981379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.981355 5607 generic.go:356] "Generic (PLEG): container finished" podID="e8d26170-cead-4921-9e17-37b97153856f" containerID="e578f3f7b96c41d0ffc69487187b532c171545f7b821cc90bf8fb1a3ec0f7e8a" exitCode=60 Oct 30 22:45:03.981477 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.981398 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" event={"ID":"e8d26170-cead-4921-9e17-37b97153856f","Type":"ContainerDied","Data":"e578f3f7b96c41d0ffc69487187b532c171545f7b821cc90bf8fb1a3ec0f7e8a"} Oct 30 22:45:03.981672 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.981654 5607 scope.go:122] "RemoveContainer" containerID="e578f3f7b96c41d0ffc69487187b532c171545f7b821cc90bf8fb1a3ec0f7e8a" Oct 30 22:45:03.981913 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:03.981894 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k_odh-ai-gateway-infra(e8d26170-cead-4921-9e17-37b97153856f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" podUID="e8d26170-cead-4921-9e17-37b97153856f" Oct 30 22:45:03.992020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:03.992002 5607 scope.go:122] "RemoveContainer" containerID="3f4f78f1bb67f17769bddd360650983cb7f5b93f0bba25841ebef32e42a8c7f2" Oct 30 22:45:04.989660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:04.989619 5607 generic.go:356] "Generic (PLEG): container finished" podID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerID="ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab" exitCode=60 Oct 30 22:45:04.990143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:04.989705 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerDied","Data":"ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab"} Oct 30 22:45:04.990143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:04.989764 5607 scope.go:122] "RemoveContainer" containerID="75520ff58cd6e3c7ef24d2ec59d2f9521f884b0b35091362481730f724b409b4" Oct 30 22:45:04.990143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:04.989902 5607 scope.go:122] "RemoveContainer" containerID="ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab" Oct 30 22:45:04.990292 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:04.990161 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2_odh-ai-gateway-infra(4a665099-6fc0-4174-8d29-ab64f4eac4b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" Oct 30 22:45:04.991814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:04.991797 5607 scope.go:122] "RemoveContainer" containerID="28ff42387c5a4b42de098f089bc5940794bf8e340a9431e7e19fa071c87a4c4e" Oct 30 22:45:04.992045 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:04.992023 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb_odh-ai-gateway-infra(b82d1693-9126-47a7-976f-21bb4a31c856)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" Oct 30 22:45:04.993390 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:04.993374 5607 scope.go:122] "RemoveContainer" containerID="e578f3f7b96c41d0ffc69487187b532c171545f7b821cc90bf8fb1a3ec0f7e8a" Oct 30 22:45:04.993554 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:04.993538 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k_odh-ai-gateway-infra(e8d26170-cead-4921-9e17-37b97153856f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" podUID="e8d26170-cead-4921-9e17-37b97153856f" Oct 30 22:45:06.003836 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:06.003812 5607 scope.go:122] "RemoveContainer" containerID="ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab" Oct 30 22:45:06.004289 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:06.004038 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2_odh-ai-gateway-infra(4a665099-6fc0-4174-8d29-ab64f4eac4b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" Oct 30 22:45:12.862856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:12.862824 5607 scope.go:122] "RemoveContainer" containerID="2ceaee3e91cf9763d46eaa690e72f1d318b16abbbf716b8234416f5808e05f6b" Oct 30 22:45:14.062404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:14.062319 5607 generic.go:356] "Generic (PLEG): container finished" podID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerID="7cb42e7ac0f4cfbf35df4e1cb96c292c98d6df172cd7dd00887096f554afa3b3" exitCode=60 Oct 30 22:45:14.062404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:14.062392 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" event={"ID":"a32fe2f2-13fc-4435-870e-c6fdc091f33d","Type":"ContainerDied","Data":"7cb42e7ac0f4cfbf35df4e1cb96c292c98d6df172cd7dd00887096f554afa3b3"} Oct 30 22:45:14.062905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:14.062452 5607 scope.go:122] "RemoveContainer" containerID="2ceaee3e91cf9763d46eaa690e72f1d318b16abbbf716b8234416f5808e05f6b" Oct 30 22:45:14.062905 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:14.062842 5607 scope.go:122] "RemoveContainer" containerID="7cb42e7ac0f4cfbf35df4e1cb96c292c98d6df172cd7dd00887096f554afa3b3" Oct 30 22:45:14.063218 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:14.063198 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b_odh-ai-gateway-infra(a32fe2f2-13fc-4435-870e-c6fdc091f33d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" Oct 30 22:45:15.197671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.197637 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b"] Oct 30 22:45:15.364358 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.364331 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:15.407814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.407780 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg\" (UniqueName: \"kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg\") pod \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " Oct 30 22:45:15.407995 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.407845 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca\") pod \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\" (UID: \"a32fe2f2-13fc-4435-870e-c6fdc091f33d\") " Oct 30 22:45:15.408236 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.408203 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca" pod "a32fe2f2-13fc-4435-870e-c6fdc091f33d" (UID: "a32fe2f2-13fc-4435-870e-c6fdc091f33d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:15.408407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.408391 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a32fe2f2-13fc-4435-870e-c6fdc091f33d-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:15.410208 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.410182 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg" pod "a32fe2f2-13fc-4435-870e-c6fdc091f33d" (UID: "a32fe2f2-13fc-4435-870e-c6fdc091f33d"). InnerVolumeSpecName "kube-api-access-c6qgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:15.509823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:15.509733 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qgg\" (UniqueName: \"kubernetes.io/projected/a32fe2f2-13fc-4435-870e-c6fdc091f33d-kube-api-access-c6qgg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:16.088660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.088587 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" event={"ID":"a32fe2f2-13fc-4435-870e-c6fdc091f33d","Type":"ContainerDied","Data":"9b8f96e1ccd674cee2fc8e4c7d817959192f2f46be0b95e86287f7e703514695"} Oct 30 22:45:16.088980 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.088943 5607 scope.go:122] "RemoveContainer" containerID="7cb42e7ac0f4cfbf35df4e1cb96c292c98d6df172cd7dd00887096f554afa3b3" Oct 30 22:45:16.089214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.089190 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b" Oct 30 22:45:16.271252 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.271217 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b"] Oct 30 22:45:16.317643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.317607 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f39eb7-29890005-pff4b"] Oct 30 22:45:16.863098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.863069 5607 scope.go:122] "RemoveContainer" containerID="28ff42387c5a4b42de098f089bc5940794bf8e340a9431e7e19fa071c87a4c4e" Oct 30 22:45:16.871835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:16.871812 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" path="/var/lib/kubelet/pods/a32fe2f2-13fc-4435-870e-c6fdc091f33d/volumes" Oct 30 22:45:17.862607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:17.862578 5607 scope.go:122] "RemoveContainer" containerID="ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab" Oct 30 22:45:18.106975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:18.106876 5607 generic.go:356] "Generic (PLEG): container finished" podID="b82d1693-9126-47a7-976f-21bb4a31c856" containerID="33c55ad26bd25e87a7f8d2876b72a130306e31e61950ffd3467a379dab7f56b0" exitCode=60 Oct 30 22:45:18.107162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:18.106989 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" event={"ID":"b82d1693-9126-47a7-976f-21bb4a31c856","Type":"ContainerDied","Data":"33c55ad26bd25e87a7f8d2876b72a130306e31e61950ffd3467a379dab7f56b0"} Oct 30 22:45:18.107162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:18.107081 5607 scope.go:122] "RemoveContainer" containerID="28ff42387c5a4b42de098f089bc5940794bf8e340a9431e7e19fa071c87a4c4e" Oct 30 22:45:18.107547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:18.107528 5607 scope.go:122] "RemoveContainer" containerID="33c55ad26bd25e87a7f8d2876b72a130306e31e61950ffd3467a379dab7f56b0" Oct 30 22:45:18.107877 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:18.107853 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb_odh-ai-gateway-infra(b82d1693-9126-47a7-976f-21bb4a31c856)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" Oct 30 22:45:18.863188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:18.863157 5607 scope.go:122] "RemoveContainer" containerID="e578f3f7b96c41d0ffc69487187b532c171545f7b821cc90bf8fb1a3ec0f7e8a" Oct 30 22:45:19.117066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.117038 5607 generic.go:356] "Generic (PLEG): container finished" podID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerID="e94b892cf252eb728c7a2b774e51341866ac21c73785c54316121c98bcb4d37f" exitCode=60 Oct 30 22:45:19.117200 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.117102 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerDied","Data":"e94b892cf252eb728c7a2b774e51341866ac21c73785c54316121c98bcb4d37f"} Oct 30 22:45:19.117200 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.117161 5607 scope.go:122] "RemoveContainer" containerID="ef2e6968a7c1907b0b12b145d305501916690b5ee42fe943934792fd721ff3ab" Oct 30 22:45:19.117596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.117578 5607 scope.go:122] "RemoveContainer" containerID="e94b892cf252eb728c7a2b774e51341866ac21c73785c54316121c98bcb4d37f" Oct 30 22:45:19.117899 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:19.117873 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2_odh-ai-gateway-infra(4a665099-6fc0-4174-8d29-ab64f4eac4b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" Oct 30 22:45:19.237601 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.237576 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb"] Oct 30 22:45:19.402694 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.402667 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:19.458506 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.458481 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca\") pod \"b82d1693-9126-47a7-976f-21bb4a31c856\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " Oct 30 22:45:19.458665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.458646 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw\" (UniqueName: \"kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw\") pod \"b82d1693-9126-47a7-976f-21bb4a31c856\" (UID: \"b82d1693-9126-47a7-976f-21bb4a31c856\") " Oct 30 22:45:19.458828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.458806 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca" pod "b82d1693-9126-47a7-976f-21bb4a31c856" (UID: "b82d1693-9126-47a7-976f-21bb4a31c856"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:19.459098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.459073 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82d1693-9126-47a7-976f-21bb4a31c856-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:19.460928 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.460902 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw" pod "b82d1693-9126-47a7-976f-21bb4a31c856" (UID: "b82d1693-9126-47a7-976f-21bb4a31c856"). InnerVolumeSpecName "kube-api-access-z5vnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:19.559808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:19.559778 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vnw\" (UniqueName: \"kubernetes.io/projected/b82d1693-9126-47a7-976f-21bb4a31c856-kube-api-access-z5vnw\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.127336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.127303 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" Oct 30 22:45:20.127887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.127304 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb" event={"ID":"b82d1693-9126-47a7-976f-21bb4a31c856","Type":"ContainerDied","Data":"d4d52520135ea80ba57fe9f4697ba5c9063368513d4effbe078fffc7eb46e806"} Oct 30 22:45:20.127887 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.127452 5607 scope.go:122] "RemoveContainer" containerID="33c55ad26bd25e87a7f8d2876b72a130306e31e61950ffd3467a379dab7f56b0" Oct 30 22:45:20.129550 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.129522 5607 generic.go:356] "Generic (PLEG): container finished" podID="e8d26170-cead-4921-9e17-37b97153856f" containerID="e0a4b31a0e2596afee23da8deba54f8209763a3c4b9df833d66779a367a895ca" exitCode=60 Oct 30 22:45:20.129651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.129587 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" event={"ID":"e8d26170-cead-4921-9e17-37b97153856f","Type":"ContainerDied","Data":"e0a4b31a0e2596afee23da8deba54f8209763a3c4b9df833d66779a367a895ca"} Oct 30 22:45:20.130076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.129997 5607 scope.go:122] "RemoveContainer" containerID="e0a4b31a0e2596afee23da8deba54f8209763a3c4b9df833d66779a367a895ca" Oct 30 22:45:20.130381 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:20.130353 5607 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k_odh-ai-gateway-infra(e8d26170-cead-4921-9e17-37b97153856f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" podUID="e8d26170-cead-4921-9e17-37b97153856f" Oct 30 22:45:20.139525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.139510 5607 scope.go:122] "RemoveContainer" containerID="e578f3f7b96c41d0ffc69487187b532c171545f7b821cc90bf8fb1a3ec0f7e8a" Oct 30 22:45:20.251008 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.250977 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2"] Oct 30 22:45:20.297372 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.297330 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb"] Oct 30 22:45:20.305797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.305748 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29890005-279bb"] Oct 30 22:45:20.418640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.418574 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:20.472297 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.472255 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca\") pod \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " Oct 30 22:45:20.472473 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.472458 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx\" (UniqueName: \"kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx\") pod \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\" (UID: \"4a665099-6fc0-4174-8d29-ab64f4eac4b7\") " Oct 30 22:45:20.472627 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.472604 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca" pod "4a665099-6fc0-4174-8d29-ab64f4eac4b7" (UID: "4a665099-6fc0-4174-8d29-ab64f4eac4b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:20.472981 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.472928 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a665099-6fc0-4174-8d29-ab64f4eac4b7-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.474926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.474902 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx" pod "4a665099-6fc0-4174-8d29-ab64f4eac4b7" (UID: "4a665099-6fc0-4174-8d29-ab64f4eac4b7"). InnerVolumeSpecName "kube-api-access-j8twx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:20.574427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.574388 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8twx\" (UniqueName: \"kubernetes.io/projected/4a665099-6fc0-4174-8d29-ab64f4eac4b7-kube-api-access-j8twx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.872863 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:20.872831 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" path="/var/lib/kubelet/pods/b82d1693-9126-47a7-976f-21bb4a31c856/volumes" Oct 30 22:45:21.142926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.142845 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" Oct 30 22:45:21.143419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.142847 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2" event={"ID":"4a665099-6fc0-4174-8d29-ab64f4eac4b7","Type":"ContainerDied","Data":"caf5ff694b269fa6ddd5f15409acc36c6c0ad1220131c3311601af9d4aa1869d"} Oct 30 22:45:21.143419 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.143021 5607 scope.go:122] "RemoveContainer" containerID="e94b892cf252eb728c7a2b774e51341866ac21c73785c54316121c98bcb4d37f" Oct 30 22:45:21.315545 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.315509 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2"] Oct 30 22:45:21.322730 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.322701 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-421286-29890005-zffd2"] Oct 30 22:45:21.437870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.437830 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k"] Oct 30 22:45:21.608891 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.608769 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:21.688490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.688406 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca\") pod \"e8d26170-cead-4921-9e17-37b97153856f\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " Oct 30 22:45:21.688663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.688555 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2\" (UniqueName: \"kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2\") pod \"e8d26170-cead-4921-9e17-37b97153856f\" (UID: \"e8d26170-cead-4921-9e17-37b97153856f\") " Oct 30 22:45:21.688802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.688777 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca" pod "e8d26170-cead-4921-9e17-37b97153856f" (UID: "e8d26170-cead-4921-9e17-37b97153856f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:21.689081 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.689061 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d26170-cead-4921-9e17-37b97153856f-maas-api-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:21.690823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.690803 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2" pod "e8d26170-cead-4921-9e17-37b97153856f" (UID: "e8d26170-cead-4921-9e17-37b97153856f"). InnerVolumeSpecName "kube-api-access-cnbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:21.790274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:21.790239 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbp2\" (UniqueName: \"kubernetes.io/projected/e8d26170-cead-4921-9e17-37b97153856f-kube-api-access-cnbp2\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:22.152944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.152866 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" Oct 30 22:45:22.153459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.152934 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k" event={"ID":"e8d26170-cead-4921-9e17-37b97153856f","Type":"ContainerDied","Data":"212660efe7ba258c9b47a054e722a4e4cf85dc61591aed39f97db0725c9b0fb8"} Oct 30 22:45:22.153459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.153065 5607 scope.go:122] "RemoveContainer" containerID="e0a4b31a0e2596afee23da8deba54f8209763a3c4b9df833d66779a367a895ca" Oct 30 22:45:22.339899 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.339866 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k"] Oct 30 22:45:22.344525 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.344503 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9a60dd-29890005-hp84k"] Oct 30 22:45:22.865335 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:22.865299 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:45:22.871966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.871927 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" path="/var/lib/kubelet/pods/4a665099-6fc0-4174-8d29-ab64f4eac4b7/volumes" Oct 30 22:45:22.872521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:22.872503 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d26170-cead-4921-9e17-37b97153856f" path="/var/lib/kubelet/pods/e8d26170-cead-4921-9e17-37b97153856f/volumes" Oct 30 22:45:51.128536 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.128489 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb"] Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129601 5607 cpu_manager.go:414] "containerMap: removing container" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129621 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129659 5607 cpu_manager.go:414] "containerMap: removing container" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129667 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129687 5607 cpu_manager.go:414] "containerMap: removing container" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129695 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129718 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129726 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129744 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129752 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129760 5607 cpu_manager.go:414] "containerMap: removing container" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129767 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129785 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129792 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129808 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129815 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129845 5607 cpu_manager.go:414] "containerMap: removing container" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129852 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129865 5607 cpu_manager.go:414] "containerMap: removing container" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.129872 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130053 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130067 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130077 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130086 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130097 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130109 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130121 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130130 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130138 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a665099-6fc0-4174-8d29-ab64f4eac4b7" containerName="cleanup" Oct 30 22:45:51.130304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.130150 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:45:51.134505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.134484 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.191803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.191765 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb"] Oct 30 22:45:51.233625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.233592 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.233805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.233650 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5slm\" (UniqueName: \"kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm\") pod \"payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.238031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.238007 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-031eee\"" Oct 30 22:45:51.238140 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.238047 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-031eee-dockercfg-4tdv9\"" Oct 30 22:45:51.335461 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.335428 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.335621 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.335497 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5slm\" (UniqueName: \"kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm\") pod \"payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.336250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.336232 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.398803 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.398741 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5slm\" (UniqueName: \"kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm\") pod \"payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.454196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.454172 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:51.841856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:51.841830 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb"] Oct 30 22:45:52.362699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:52.362670 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" event={"ID":"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d","Type":"ContainerStarted","Data":"2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e"} Oct 30 22:45:52.363158 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:52.362719 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" event={"ID":"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d","Type":"ContainerStarted","Data":"59bf9538d85614c2abb816f18203d09b8110aad4ff10ca554a4ae8e44a874f94"} Oct 30 22:45:52.363158 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:52.362786 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:45:52.484008 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:45:52.483944 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" podStartSLOduration=1.483931085 podStartE2EDuration="1.483931085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:52.480664444 +0000 UTC m=+3531.881220567" watchObservedRunningTime="2026-10-30 22:45:52.483931085 +0000 UTC m=+3531.884487205" Oct 30 22:45:52.864867 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:45:52.864747 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:46:03.373014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:03.372984 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:46:22.866837 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:46:22.866804 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:46:35.401611 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.401570 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n"] Oct 30 22:46:35.402332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.402316 5607 cpu_manager.go:414] "containerMap: removing container" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:46:35.402384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.402335 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82d1693-9126-47a7-976f-21bb4a31c856" containerName="cleanup" Oct 30 22:46:35.402384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.402370 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:46:35.402384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.402375 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:46:35.402492 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.402482 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d26170-cead-4921-9e17-37b97153856f" containerName="cleanup" Oct 30 22:46:35.402533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.402495 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32fe2f2-13fc-4435-870e-c6fdc091f33d" containerName="cleanup" Oct 30 22:46:35.406303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.406284 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.515913 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.515871 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjjh\" (UniqueName: \"kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh\") pod \"payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.516384 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.516157 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.517308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.517284 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-293c80\"" Oct 30 22:46:35.519330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.519295 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-293c80-dockercfg-8v9j7\"" Oct 30 22:46:35.523771 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.523747 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n"] Oct 30 22:46:35.542823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.542791 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8"] Oct 30 22:46:35.550004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.549943 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:35.605811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.605788 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8"] Oct 30 22:46:35.617155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.617126 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.617309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.617201 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjjh\" (UniqueName: \"kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh\") pod \"payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.618118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.618095 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.680431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.680367 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjjh\" (UniqueName: \"kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh\") pod \"payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.718292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.718265 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkpl\" (UniqueName: \"kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl\") pod \"payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:35.718410 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.718336 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:35.720893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.720870 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:35.820053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.820019 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkpl\" (UniqueName: \"kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl\") pod \"payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:35.820225 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.820153 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:35.820912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.820887 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:35.883115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:35.883084 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkpl\" (UniqueName: \"kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl\") pod \"payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:36.093858 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:46:36.093829 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e411daa_bada_4599_9c96_38e1dc7d1bac.slice/crio-93b4bd3d5a6340f82b0030cce0942dfdbecad30be5ff3da41b2f03fba41dcb6c WatchSource:0}: Error finding container 93b4bd3d5a6340f82b0030cce0942dfdbecad30be5ff3da41b2f03fba41dcb6c: Status 404 returned error can't find the container with id 93b4bd3d5a6340f82b0030cce0942dfdbecad30be5ff3da41b2f03fba41dcb6c Oct 30 22:46:36.095819 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.095727 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n"] Oct 30 22:46:36.167413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.167388 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:36.539610 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:46:36.539585 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37711f92_7c68_4c03_97c8_c86d2e012007.slice/crio-1ddd2aa965465a839fecc8b547b17e2e1890ee2821033e96be63c63b9e78de7f WatchSource:0}: Error finding container 1ddd2aa965465a839fecc8b547b17e2e1890ee2821033e96be63c63b9e78de7f: Status 404 returned error can't find the container with id 1ddd2aa965465a839fecc8b547b17e2e1890ee2821033e96be63c63b9e78de7f Oct 30 22:46:36.542514 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.542491 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8"] Oct 30 22:46:36.679637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.679598 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" event={"ID":"37711f92-7c68-4c03-97c8-c86d2e012007","Type":"ContainerStarted","Data":"0511d61a1769d76bc2dc35e9d87dec7053dc9f5e8e3f648add6a348f4dadbc6d"} Oct 30 22:46:36.679783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.679663 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" event={"ID":"37711f92-7c68-4c03-97c8-c86d2e012007","Type":"ContainerStarted","Data":"1ddd2aa965465a839fecc8b547b17e2e1890ee2821033e96be63c63b9e78de7f"} Oct 30 22:46:36.679783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.679761 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:36.681535 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.681511 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" event={"ID":"8e411daa-bada-4599-9c96-38e1dc7d1bac","Type":"ContainerStarted","Data":"b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd"} Oct 30 22:46:36.681659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.681563 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:36.681659 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.681587 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" event={"ID":"8e411daa-bada-4599-9c96-38e1dc7d1bac","Type":"ContainerStarted","Data":"93b4bd3d5a6340f82b0030cce0942dfdbecad30be5ff3da41b2f03fba41dcb6c"} Oct 30 22:46:36.800939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.800856 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" podStartSLOduration=1.8008427729999998 podStartE2EDuration="1.800842773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:36.797553357 +0000 UTC m=+3576.198109477" watchObservedRunningTime="2026-10-30 22:46:36.800842773 +0000 UTC m=+3576.201398894" Oct 30 22:46:36.916855 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:36.916810 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" podStartSLOduration=1.916797069 podStartE2EDuration="1.916797069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:36.914470307 +0000 UTC m=+3576.315026429" watchObservedRunningTime="2026-10-30 22:46:36.916797069 +0000 UTC m=+3576.317353189" Oct 30 22:46:47.692039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:47.692009 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:46:47.692486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:46:47.692153 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:46:52.867712 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:46:52.867649 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:47:01.415037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:01.415005 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:47:01.420864 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:01.420842 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:47:20.024258 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.024181 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6"] Oct 30 22:47:20.029145 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.029122 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.127431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.127390 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.127612 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.127452 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.127664 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.127611 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pqj\" (UniqueName: \"kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.127769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.127751 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.135142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.135115 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-031eee-serving-cert\"" Oct 30 22:47:20.138407 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.138380 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6"] Oct 30 22:47:20.229306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.229272 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.229521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.229350 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.229521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.229379 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.229521 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:47:20.229440 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-serving-cert: secret "maas-api-e2e-extmod-a-031eee-serving-cert" not found Oct 30 22:47:20.229521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.229460 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pqj\" (UniqueName: \"kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.229521 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:47:20.229516 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls podName:bbac6610-8829-4d6e-81c8-994755287db3 nodeName:}" failed. No retries permitted until 2026-10-30 22:47:20.72949429 +0000 UTC m=+3620.130050408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls") pod "maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" (UID: "bbac6610-8829-4d6e-81c8-994755287db3") : secret "maas-api-e2e-extmod-a-031eee-serving-cert" not found Oct 30 22:47:20.230198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.230178 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.232114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.232087 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.292547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.292478 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pqj\" (UniqueName: \"kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.737300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.737258 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.739914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.739892 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:20.947207 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:20.947173 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:21.325358 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:47:21.325329 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbac6610_8829_4d6e_81c8_994755287db3.slice/crio-a3a6df07966775698062af10621804aadc5978e536586ad182e68cb02e78fbf9 WatchSource:0}: Error finding container a3a6df07966775698062af10621804aadc5978e536586ad182e68cb02e78fbf9: Status 404 returned error can't find the container with id a3a6df07966775698062af10621804aadc5978e536586ad182e68cb02e78fbf9 Oct 30 22:47:21.330337 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:21.330310 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6"] Oct 30 22:47:22.000676 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:22.000639 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" event={"ID":"bbac6610-8829-4d6e-81c8-994755287db3","Type":"ContainerStarted","Data":"a3a6df07966775698062af10621804aadc5978e536586ad182e68cb02e78fbf9"} Oct 30 22:47:22.863911 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:47:22.863873 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:47:23.009076 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:23.009038 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" event={"ID":"bbac6610-8829-4d6e-81c8-994755287db3","Type":"ContainerStarted","Data":"e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd"} Oct 30 22:47:23.009309 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:23.009109 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:23.131744 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:23.131696 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" podStartSLOduration=3.156669158 podStartE2EDuration="4.131681371s" totalImagesPullingTime="975.012213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:23.129722154 +0000 UTC m=+3622.530278272" watchObservedRunningTime="2026-10-30 22:47:23.131681371 +0000 UTC m=+3622.532237536" Oct 30 22:47:29.021143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:29.021109 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:47:35.572632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:35.572590 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq"] Oct 30 22:47:40.573700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.573652 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" podUID="ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" containerName="payload-pre-processing" containerID="cri-o://1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3" gracePeriod=25 Oct 30 22:47:40.834899 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.834846 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:47:40.888184 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.888150 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume\") pod \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " Oct 30 22:47:40.888360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.888235 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh\" (UniqueName: \"kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh\") pod \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\" (UID: \"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc\") " Oct 30 22:47:40.888565 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.888525 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume" pod "ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" (UID: "ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:40.888865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.888848 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:40.890524 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.890500 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh" pod "ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" (UID: "ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc"). InnerVolumeSpecName "kube-api-access-jvtfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:40.989871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:40.989836 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtfh\" (UniqueName: \"kubernetes.io/projected/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc-kube-api-access-jvtfh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:41.132244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.132167 5607 generic.go:356] "Generic (PLEG): container finished" podID="ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" containerID="1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3" exitCode=0 Oct 30 22:47:41.132244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.132214 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" event={"ID":"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc","Type":"ContainerDied","Data":"1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3"} Oct 30 22:47:41.132244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.132232 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" Oct 30 22:47:41.132460 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.132293 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq" event={"ID":"ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc","Type":"ContainerDied","Data":"b3779044d52264daab0150e4a5e65a0019ee37db2ed2209ac23c42c43909a6de"} Oct 30 22:47:41.132460 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.132325 5607 scope.go:122] "RemoveContainer" containerID="1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3" Oct 30 22:47:41.143467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.143448 5607 scope.go:122] "RemoveContainer" containerID="1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3" Oct 30 22:47:41.143761 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:47:41.143729 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3\": container with ID starting with 1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3 not found: ID does not exist" containerID="1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3" Oct 30 22:47:41.143861 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.143770 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3"} err="failed to get container status \"1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3\": rpc error: code = NotFound desc = could not find container \"1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3\": container with ID starting with 1976a79121d35d463df080e11bef4a1a511133d0a574c8e006fd755a5fcd5ae3 not found: ID does not exist" Oct 30 22:47:41.303153 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.303123 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq"] Oct 30 22:47:41.307794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:41.307774 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-dcde03-b87bd6bdc-7bjtq"] Oct 30 22:47:42.871272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:47:42.871232 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" path="/var/lib/kubelet/pods/ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc/volumes" Oct 30 22:47:52.864052 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:47:52.864015 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:48:22.869384 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:48:22.869344 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:48:52.866022 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:48:52.865981 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:48:57.124417 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:48:57.124378 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb"] Oct 30 22:49:02.125054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.125002 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" podUID="2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" containerName="payload-processing" containerID="cri-o://2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e" gracePeriod=25 Oct 30 22:49:02.416197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.416172 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:49:02.524474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.524448 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm\" (UniqueName: \"kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm\") pod \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " Oct 30 22:49:02.524640 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.524493 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume\") pod \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\" (UID: \"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d\") " Oct 30 22:49:02.524888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.524866 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume" pod "2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" (UID: "2a3e1a0b-7d10-458a-85dc-22f0fb896b9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:02.525087 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.525068 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:02.526816 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.526793 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm" pod "2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" (UID: "2a3e1a0b-7d10-458a-85dc-22f0fb896b9d"). InnerVolumeSpecName "kube-api-access-h5slm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:02.626631 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.626596 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5slm\" (UniqueName: \"kubernetes.io/projected/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d-kube-api-access-h5slm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:02.699182 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.699152 5607 generic.go:356] "Generic (PLEG): container finished" podID="2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" containerID="2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e" exitCode=0 Oct 30 22:49:02.699338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.699198 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" Oct 30 22:49:02.699338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.699242 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" event={"ID":"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d","Type":"ContainerDied","Data":"2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e"} Oct 30 22:49:02.699338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.699318 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb" event={"ID":"2a3e1a0b-7d10-458a-85dc-22f0fb896b9d","Type":"ContainerDied","Data":"59bf9538d85614c2abb816f18203d09b8110aad4ff10ca554a4ae8e44a874f94"} Oct 30 22:49:02.699472 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.699341 5607 scope.go:122] "RemoveContainer" containerID="2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e" Oct 30 22:49:02.709858 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.709842 5607 scope.go:122] "RemoveContainer" containerID="2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e" Oct 30 22:49:02.710165 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:02.710135 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e\": container with ID starting with 2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e not found: ID does not exist" containerID="2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e" Oct 30 22:49:02.710262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.710173 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e"} err="failed to get container status \"2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e\": rpc error: code = NotFound desc = could not find container \"2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e\": container with ID starting with 2413c1489e78c14d2b2f51c639454bf8a1253e0bc11dfad44f836d24ca828c6e not found: ID does not exist" Oct 30 22:49:02.872276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.872248 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb"] Oct 30 22:49:02.874241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.874217 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-031eee-69f7cb7fd5-tnckb"] Oct 30 22:49:02.984184 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.984122 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6"] Oct 30 22:49:02.984371 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:02.984330 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" podUID="bbac6610-8829-4d6e-81c8-994755287db3" containerName="maas-api" containerID="cri-o://e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd" gracePeriod=30 Oct 30 22:49:03.258307 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.258285 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:49:03.335716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.335686 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca\") pod \"bbac6610-8829-4d6e-81c8-994755287db3\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " Oct 30 22:49:03.335856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.335739 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj\" (UniqueName: \"kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj\") pod \"bbac6610-8829-4d6e-81c8-994755287db3\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " Oct 30 22:49:03.335856 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.335830 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\") pod \"bbac6610-8829-4d6e-81c8-994755287db3\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " Oct 30 22:49:03.336014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.335871 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs\") pod \"bbac6610-8829-4d6e-81c8-994755287db3\" (UID: \"bbac6610-8829-4d6e-81c8-994755287db3\") " Oct 30 22:49:03.336082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.336051 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca" pod "bbac6610-8829-4d6e-81c8-994755287db3" (UID: "bbac6610-8829-4d6e-81c8-994755287db3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:03.336382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.336354 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbac6610-8829-4d6e-81c8-994755287db3-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:03.338077 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.338037 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj" pod "bbac6610-8829-4d6e-81c8-994755287db3" (UID: "bbac6610-8829-4d6e-81c8-994755287db3"). InnerVolumeSpecName "kube-api-access-d9pqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:03.338168 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.338122 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls" pod "bbac6610-8829-4d6e-81c8-994755287db3" (UID: "bbac6610-8829-4d6e-81c8-994755287db3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:03.338220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.338169 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs" pod "bbac6610-8829-4d6e-81c8-994755287db3" (UID: "bbac6610-8829-4d6e-81c8-994755287db3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:03.437260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.437221 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pqj\" (UniqueName: \"kubernetes.io/projected/bbac6610-8829-4d6e-81c8-994755287db3-kube-api-access-d9pqj\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:03.437260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.437255 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:03.437260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.437265 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbac6610-8829-4d6e-81c8-994755287db3-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:03.708195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.708161 5607 generic.go:356] "Generic (PLEG): container finished" podID="bbac6610-8829-4d6e-81c8-994755287db3" containerID="e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd" exitCode=0 Oct 30 22:49:03.708385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.708209 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" Oct 30 22:49:03.708385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.708227 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" event={"ID":"bbac6610-8829-4d6e-81c8-994755287db3","Type":"ContainerDied","Data":"e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd"} Oct 30 22:49:03.708385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.708282 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6" event={"ID":"bbac6610-8829-4d6e-81c8-994755287db3","Type":"ContainerDied","Data":"a3a6df07966775698062af10621804aadc5978e536586ad182e68cb02e78fbf9"} Oct 30 22:49:03.708385 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.708311 5607 scope.go:122] "RemoveContainer" containerID="e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd" Oct 30 22:49:03.719348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.719331 5607 scope.go:122] "RemoveContainer" containerID="e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd" Oct 30 22:49:03.719598 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:03.719578 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd\": container with ID starting with e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd not found: ID does not exist" containerID="e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd" Oct 30 22:49:03.719695 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.719606 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd"} err="failed to get container status \"e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd\": rpc error: code = NotFound desc = could not find container \"e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd\": container with ID starting with e5860a45fb6278f4c3d620b69bc8efcbd3c45744cf23f725d3e0b4b4636c02cd not found: ID does not exist" Oct 30 22:49:03.878459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.878434 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6"] Oct 30 22:49:03.884032 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:03.884011 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-031eee-588fc79f79-m96x6"] Oct 30 22:49:04.876643 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:04.876598 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" path="/var/lib/kubelet/pods/2a3e1a0b-7d10-458a-85dc-22f0fb896b9d/volumes" Oct 30 22:49:04.877263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:04.877244 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbac6610-8829-4d6e-81c8-994755287db3" path="/var/lib/kubelet/pods/bbac6610-8829-4d6e-81c8-994755287db3/volumes" Oct 30 22:49:08.624742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:08.624708 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85"] Oct 30 22:49:08.625221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:08.624921 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" containerID="cri-o://47ebefc2061faeb33b34b74f8357a0f2b5e1b1fdd6860577b1377f89f84a1ac0" gracePeriod=30 Oct 30 22:49:13.068142 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:13.068098 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:49:13.068665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:13.068165 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:49:13.779721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:13.779692 5607 generic.go:356] "Generic (PLEG): container finished" podID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerID="47ebefc2061faeb33b34b74f8357a0f2b5e1b1fdd6860577b1377f89f84a1ac0" exitCode=0 Oct 30 22:49:13.779872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:13.779760 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" event={"ID":"51e967b0-69dd-46e9-99e5-796a073a73d8","Type":"ContainerDied","Data":"47ebefc2061faeb33b34b74f8357a0f2b5e1b1fdd6860577b1377f89f84a1ac0"} Oct 30 22:49:13.928479 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:13.928448 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:49:14.058701 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058672 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.058871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058726 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.058871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058783 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.058871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058811 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.058871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058859 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.059090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058916 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.059090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058933 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:14.059090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.058984 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.059090 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059052 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.059319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059098 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:14.059319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059159 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy\") pod \"51e967b0-69dd-46e9-99e5-796a073a73d8\" (UID: \"51e967b0-69dd-46e9-99e5-796a073a73d8\") " Oct 30 22:49:14.059319 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059177 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:14.059499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059299 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:14.059499 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059390 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:14.059686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059665 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e967b0-69dd-46e9-99e5-796a073a73d8-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.059747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059692 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.059747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059710 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.059747 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059725 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.059879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.059801 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.061277 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.061253 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:14.061398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.061369 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "kube-api-access-dlspr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:14.061713 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.061692 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:49:14.061752 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.061724 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy" pod "51e967b0-69dd-46e9-99e5-796a073a73d8" (UID: "51e967b0-69dd-46e9-99e5-796a073a73d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:14.161662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.161586 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.161662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.161612 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlspr\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-kube-api-access-dlspr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.161662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.161635 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.161662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.161648 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e967b0-69dd-46e9-99e5-796a073a73d8-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:14.788021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.787976 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" event={"ID":"51e967b0-69dd-46e9-99e5-796a073a73d8","Type":"ContainerDied","Data":"a3a4ff0dbe8e16a2f40a4b2b90ecc37d6e565432f7716058246869fe14fb4171"} Oct 30 22:49:14.788226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.788034 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85" Oct 30 22:49:14.788226 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.788076 5607 scope.go:122] "RemoveContainer" containerID="47ebefc2061faeb33b34b74f8357a0f2b5e1b1fdd6860577b1377f89f84a1ac0" Oct 30 22:49:14.965022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:14.964987 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85"] Oct 30 22:49:15.015966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:15.015923 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-c7k85"] Oct 30 22:49:16.871097 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:16.871064 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" path="/var/lib/kubelet/pods/51e967b0-69dd-46e9-99e5-796a073a73d8/volumes" Oct 30 22:49:19.950256 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:19.950220 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q"] Oct 30 22:49:20.001118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:20.001090 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz"] Oct 30 22:49:22.865283 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:22.865243 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:49:27.199109 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:27.199075 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8"] Oct 30 22:49:27.339048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:27.339014 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb"] Oct 30 22:49:32.200280 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:32.200239 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerName="payload-processing" containerID="cri-o://d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b" gracePeriod=25 Oct 30 22:49:32.339754 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:32.339717 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerName="payload-pre-processing" containerID="cri-o://39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7" gracePeriod=25 Oct 30 22:49:34.972711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:34.972672 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" containerID="cri-o://7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0" gracePeriod=45 Oct 30 22:49:35.022781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.022751 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" containerID="cri-o://120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59" gracePeriod=45 Oct 30 22:49:35.318591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.318565 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:49:35.322544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.322526 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:49:35.398834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.398798 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache\") pod \"12f8222d-6aac-4906-aba3-54f1ecb79165\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " Oct 30 22:49:35.399029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.398901 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r\" (UniqueName: \"kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r\") pod \"12f8222d-6aac-4906-aba3-54f1ecb79165\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " Oct 30 22:49:35.399108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399065 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache" pod "12f8222d-6aac-4906-aba3-54f1ecb79165" (UID: "12f8222d-6aac-4906-aba3-54f1ecb79165"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.399178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399161 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir\") pod \"98f3e9be-0d98-419d-8bb0-8309858d897a\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " Oct 30 22:49:35.399251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399163 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir" pod "98f3e9be-0d98-419d-8bb0-8309858d897a" (UID: "98f3e9be-0d98-419d-8bb0-8309858d897a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.399304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399262 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm\") pod \"98f3e9be-0d98-419d-8bb0-8309858d897a\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " Oct 30 22:49:35.399360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399317 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs\" (UniqueName: \"kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs\") pod \"12f8222d-6aac-4906-aba3-54f1ecb79165\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " Oct 30 22:49:35.399360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399337 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm\") pod \"12f8222d-6aac-4906-aba3-54f1ecb79165\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " Oct 30 22:49:35.399480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399359 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir\") pod \"12f8222d-6aac-4906-aba3-54f1ecb79165\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " Oct 30 22:49:35.399480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399391 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache\") pod \"98f3e9be-0d98-419d-8bb0-8309858d897a\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " Oct 30 22:49:35.399480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399427 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home\") pod \"98f3e9be-0d98-419d-8bb0-8309858d897a\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " Oct 30 22:49:35.399480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399470 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home\") pod \"12f8222d-6aac-4906-aba3-54f1ecb79165\" (UID: \"12f8222d-6aac-4906-aba3-54f1ecb79165\") " Oct 30 22:49:35.399705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399496 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447\" (UniqueName: \"kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447\") pod \"98f3e9be-0d98-419d-8bb0-8309858d897a\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " Oct 30 22:49:35.399705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399591 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs\" (UniqueName: \"kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs\") pod \"98f3e9be-0d98-419d-8bb0-8309858d897a\" (UID: \"98f3e9be-0d98-419d-8bb0-8309858d897a\") " Oct 30 22:49:35.399908 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.399829 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache" pod "98f3e9be-0d98-419d-8bb0-8309858d897a" (UID: "98f3e9be-0d98-419d-8bb0-8309858d897a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.400284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.400252 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir" pod "12f8222d-6aac-4906-aba3-54f1ecb79165" (UID: "12f8222d-6aac-4906-aba3-54f1ecb79165"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.400404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.400284 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.400404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.400304 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.400404 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.400320 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.400556 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.400396 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home" pod "98f3e9be-0d98-419d-8bb0-8309858d897a" (UID: "98f3e9be-0d98-419d-8bb0-8309858d897a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.400614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.400573 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home" pod "12f8222d-6aac-4906-aba3-54f1ecb79165" (UID: "12f8222d-6aac-4906-aba3-54f1ecb79165"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.402757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.402734 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r" pod "12f8222d-6aac-4906-aba3-54f1ecb79165" (UID: "12f8222d-6aac-4906-aba3-54f1ecb79165"). InnerVolumeSpecName "kube-api-access-6n24r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.403374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.403348 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs" pod "98f3e9be-0d98-419d-8bb0-8309858d897a" (UID: "98f3e9be-0d98-419d-8bb0-8309858d897a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.403538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.403411 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm" pod "12f8222d-6aac-4906-aba3-54f1ecb79165" (UID: "12f8222d-6aac-4906-aba3-54f1ecb79165"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.403538 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.403447 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447" pod "98f3e9be-0d98-419d-8bb0-8309858d897a" (UID: "98f3e9be-0d98-419d-8bb0-8309858d897a"). InnerVolumeSpecName "kube-api-access-kx447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.403635 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.403539 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs" pod "12f8222d-6aac-4906-aba3-54f1ecb79165" (UID: "12f8222d-6aac-4906-aba3-54f1ecb79165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.403901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.403881 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm" pod "98f3e9be-0d98-419d-8bb0-8309858d897a" (UID: "98f3e9be-0d98-419d-8bb0-8309858d897a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501849 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n24r\" (UniqueName: \"kubernetes.io/projected/12f8222d-6aac-4906-aba3-54f1ecb79165-kube-api-access-6n24r\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501880 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501891 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f8222d-6aac-4906-aba3-54f1ecb79165-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501898 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501906 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501913 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f3e9be-0d98-419d-8bb0-8309858d897a-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.501917 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501921 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f8222d-6aac-4906-aba3-54f1ecb79165-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.502399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501929 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx447\" (UniqueName: \"kubernetes.io/projected/98f3e9be-0d98-419d-8bb0-8309858d897a-kube-api-access-kx447\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.502399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.501943 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f3e9be-0d98-419d-8bb0-8309858d897a-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.930610 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.930527 5607 generic.go:356] "Generic (PLEG): container finished" podID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerID="120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59" exitCode=0 Oct 30 22:49:35.930769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.930620 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" Oct 30 22:49:35.930769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.930676 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" event={"ID":"12f8222d-6aac-4906-aba3-54f1ecb79165","Type":"ContainerDied","Data":"120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59"} Oct 30 22:49:35.930769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.930712 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz" event={"ID":"12f8222d-6aac-4906-aba3-54f1ecb79165","Type":"ContainerDied","Data":"2a1333effc045b535c7289029e1a701654b0740767903b009756da5cb10be32a"} Oct 30 22:49:35.930769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.930734 5607 scope.go:122] "RemoveContainer" containerID="120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59" Oct 30 22:49:35.932171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.932150 5607 generic.go:356] "Generic (PLEG): container finished" podID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerID="7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0" exitCode=0 Oct 30 22:49:35.932282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.932191 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" Oct 30 22:49:35.932282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.932232 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" event={"ID":"98f3e9be-0d98-419d-8bb0-8309858d897a","Type":"ContainerDied","Data":"7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0"} Oct 30 22:49:35.932282 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.932269 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q" event={"ID":"98f3e9be-0d98-419d-8bb0-8309858d897a","Type":"ContainerDied","Data":"94b400a02a2f3fbd935ec3b4726e478b566c61d6dc2bc073514b884a14f575c2"} Oct 30 22:49:35.945707 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.945690 5607 scope.go:122] "RemoveContainer" containerID="120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59" Oct 30 22:49:35.946008 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:35.945988 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59\": container with ID starting with 120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59 not found: ID does not exist" containerID="120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59" Oct 30 22:49:35.946095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.946017 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59"} err="failed to get container status \"120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59\": rpc error: code = NotFound desc = could not find container \"120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59\": container with ID starting with 120cf8ea456f519d0fb9b27601bcaf197be348dcf8408c2470761ed97b678d59 not found: ID does not exist" Oct 30 22:49:35.946095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.946035 5607 scope.go:122] "RemoveContainer" containerID="7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0" Oct 30 22:49:35.955802 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.955784 5607 scope.go:122] "RemoveContainer" containerID="7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0" Oct 30 22:49:35.956092 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:35.956073 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0\": container with ID starting with 7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0 not found: ID does not exist" containerID="7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0" Oct 30 22:49:35.956166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:35.956100 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0"} err="failed to get container status \"7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0\": rpc error: code = NotFound desc = could not find container \"7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0\": container with ID starting with 7cc8da761cf0b5aa15872cf7893213c547ec99976e743f4c34c10afc6192a4b0 not found: ID does not exist" Oct 30 22:49:36.099513 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:36.099373 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz"] Oct 30 22:49:36.102203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:36.102180 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/premium-w1-e65019-kserve-56d658b4f7-7m7zz"] Oct 30 22:49:36.214038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:36.214017 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q"] Oct 30 22:49:36.218043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:36.218022 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/distinct-w1-e65019-kserve-5d854f576d-sb25q"] Oct 30 22:49:36.871568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:36.871536 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" path="/var/lib/kubelet/pods/12f8222d-6aac-4906-aba3-54f1ecb79165/volumes" Oct 30 22:49:36.871973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:36.871944 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" path="/var/lib/kubelet/pods/98f3e9be-0d98-419d-8bb0-8309858d897a/volumes" Oct 30 22:49:40.043662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:40.043627 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:49:40.044118 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:40.043788 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:49:43.247557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:43.247526 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p"] Oct 30 22:49:43.360049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:43.360016 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq"] Oct 30 22:49:48.247976 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.247900 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" podUID="48bf32a3-efed-4c9a-bbd1-64815f1b60ec" containerName="payload-processing" containerID="cri-o://352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522" gracePeriod=25 Oct 30 22:49:48.360586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.360559 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" podUID="805fd112-9e46-4ac0-b482-5aab9c6ec1b8" containerName="payload-pre-processing" containerID="cri-o://fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2" gracePeriod=25 Oct 30 22:49:48.509588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.509534 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:49:48.640881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.640851 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume\") pod \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " Oct 30 22:49:48.641082 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.640910 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm\" (UniqueName: \"kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm\") pod \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\" (UID: \"48bf32a3-efed-4c9a-bbd1-64815f1b60ec\") " Oct 30 22:49:48.641705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.641400 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume" pod "48bf32a3-efed-4c9a-bbd1-64815f1b60ec" (UID: "48bf32a3-efed-4c9a-bbd1-64815f1b60ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:48.641831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.641744 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:48.644211 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.644189 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm" pod "48bf32a3-efed-4c9a-bbd1-64815f1b60ec" (UID: "48bf32a3-efed-4c9a-bbd1-64815f1b60ec"). InnerVolumeSpecName "kube-api-access-sptdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:48.645827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.645806 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:49:48.743145 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.743121 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume\") pod \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " Oct 30 22:49:48.743285 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.743235 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44\" (UniqueName: \"kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44\") pod \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\" (UID: \"805fd112-9e46-4ac0-b482-5aab9c6ec1b8\") " Oct 30 22:49:48.743496 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.743484 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sptdm\" (UniqueName: \"kubernetes.io/projected/48bf32a3-efed-4c9a-bbd1-64815f1b60ec-kube-api-access-sptdm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:48.743576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.743513 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume" pod "805fd112-9e46-4ac0-b482-5aab9c6ec1b8" (UID: "805fd112-9e46-4ac0-b482-5aab9c6ec1b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:48.745388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.745370 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44" pod "805fd112-9e46-4ac0-b482-5aab9c6ec1b8" (UID: "805fd112-9e46-4ac0-b482-5aab9c6ec1b8"). InnerVolumeSpecName "kube-api-access-rmf44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:48.844563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.844501 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmf44\" (UniqueName: \"kubernetes.io/projected/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-kube-api-access-rmf44\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:48.844563 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:48.844525 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805fd112-9e46-4ac0-b482-5aab9c6ec1b8-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:49.028726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.028699 5607 generic.go:356] "Generic (PLEG): container finished" podID="48bf32a3-efed-4c9a-bbd1-64815f1b60ec" containerID="352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522" exitCode=0 Oct 30 22:49:49.028846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.028755 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" Oct 30 22:49:49.028846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.028786 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" event={"ID":"48bf32a3-efed-4c9a-bbd1-64815f1b60ec","Type":"ContainerDied","Data":"352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522"} Oct 30 22:49:49.028846 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.028831 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p" event={"ID":"48bf32a3-efed-4c9a-bbd1-64815f1b60ec","Type":"ContainerDied","Data":"cdd8759fb662b1df1fd550cae0d829dcb308a78a025dea7370c323f82d7c4aeb"} Oct 30 22:49:49.029022 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.028859 5607 scope.go:122] "RemoveContainer" containerID="352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522" Oct 30 22:49:49.030475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.030453 5607 generic.go:356] "Generic (PLEG): container finished" podID="805fd112-9e46-4ac0-b482-5aab9c6ec1b8" containerID="fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2" exitCode=0 Oct 30 22:49:49.030595 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.030565 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" Oct 30 22:49:49.030729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.030698 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" event={"ID":"805fd112-9e46-4ac0-b482-5aab9c6ec1b8","Type":"ContainerDied","Data":"fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2"} Oct 30 22:49:49.030815 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.030744 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq" event={"ID":"805fd112-9e46-4ac0-b482-5aab9c6ec1b8","Type":"ContainerDied","Data":"62a015b03fb54e626ce7384c1f6985c691d020c9fec58bf201e6eedd780289b5"} Oct 30 22:49:49.039645 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.039627 5607 scope.go:122] "RemoveContainer" containerID="352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522" Oct 30 22:49:49.039912 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:49.039891 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522\": container with ID starting with 352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522 not found: ID does not exist" containerID="352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522" Oct 30 22:49:49.040100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.039919 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522"} err="failed to get container status \"352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522\": rpc error: code = NotFound desc = could not find container \"352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522\": container with ID starting with 352141ba89fc55b5d07bbd4a2336e18112f872f7c011412e350f9425dbbfa522 not found: ID does not exist" Oct 30 22:49:49.040100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.039936 5607 scope.go:122] "RemoveContainer" containerID="fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2" Oct 30 22:49:49.050670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.050639 5607 scope.go:122] "RemoveContainer" containerID="fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2" Oct 30 22:49:49.050965 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:49.050933 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2\": container with ID starting with fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2 not found: ID does not exist" containerID="fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2" Oct 30 22:49:49.051042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.050976 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2"} err="failed to get container status \"fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2\": rpc error: code = NotFound desc = could not find container \"fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2\": container with ID starting with fd3f25ada072d227c52a97043e56baf2c743e48ed5580c958a1c195b69043cc2 not found: ID does not exist" Oct 30 22:49:49.198450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.198425 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p"] Oct 30 22:49:49.205458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.205440 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9a60dd-79f5bb4d9f-9qb9p"] Oct 30 22:49:49.320216 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.320192 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq"] Oct 30 22:49:49.324458 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:49.324438 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9a60dd-68b9bd86c-whklq"] Oct 30 22:49:50.043370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:50.043325 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:49:50.043580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:50.043338 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:49:50.870751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:50.870720 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bf32a3-efed-4c9a-bbd1-64815f1b60ec" path="/var/lib/kubelet/pods/48bf32a3-efed-4c9a-bbd1-64815f1b60ec/volumes" Oct 30 22:49:50.871126 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:50.871110 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805fd112-9e46-4ac0-b482-5aab9c6ec1b8" path="/var/lib/kubelet/pods/805fd112-9e46-4ac0-b482-5aab9c6ec1b8/volumes" Oct 30 22:49:52.484299 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.484274 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:49:52.582325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.582295 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr\" (UniqueName: \"kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr\") pod \"36ca4a4f-814f-4467-850f-7c72ca21d55c\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " Oct 30 22:49:52.582481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.582464 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume\") pod \"36ca4a4f-814f-4467-850f-7c72ca21d55c\" (UID: \"36ca4a4f-814f-4467-850f-7c72ca21d55c\") " Oct 30 22:49:52.582895 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.582871 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume" pod "36ca4a4f-814f-4467-850f-7c72ca21d55c" (UID: "36ca4a4f-814f-4467-850f-7c72ca21d55c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:52.584524 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.584497 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr" pod "36ca4a4f-814f-4467-850f-7c72ca21d55c" (UID: "36ca4a4f-814f-4467-850f-7c72ca21d55c"). InnerVolumeSpecName "kube-api-access-8lccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:52.603412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.603386 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:49:52.684270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.684243 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume\") pod \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " Oct 30 22:49:52.684396 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.684279 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng\" (UniqueName: \"kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng\") pod \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\" (UID: \"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f\") " Oct 30 22:49:52.684603 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.684582 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume" pod "92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" (UID: "92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:52.684820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.684793 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lccr\" (UniqueName: \"kubernetes.io/projected/36ca4a4f-814f-4467-850f-7c72ca21d55c-kube-api-access-8lccr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:52.684820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.684819 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:52.685157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.684830 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ca4a4f-814f-4467-850f-7c72ca21d55c-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:52.686561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.686540 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng" pod "92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" (UID: "92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f"). InnerVolumeSpecName "kube-api-access-4z2ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:52.786488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:52.786424 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2ng\" (UniqueName: \"kubernetes.io/projected/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f-kube-api-access-4z2ng\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:52.863266 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:52.863235 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:49:53.061542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.061483 5607 generic.go:356] "Generic (PLEG): container finished" podID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerID="39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7" exitCode=0 Oct 30 22:49:53.061542 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.061529 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" Oct 30 22:49:53.061781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.061570 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" event={"ID":"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f","Type":"ContainerDied","Data":"39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7"} Oct 30 22:49:53.061781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.061624 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb" event={"ID":"92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f","Type":"ContainerDied","Data":"fbcd58a94d2e5df63eee50bb848024e68c81fd5044a9445ad4e28091954a2019"} Oct 30 22:49:53.061781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.061651 5607 scope.go:122] "RemoveContainer" containerID="39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7" Oct 30 22:49:53.063241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.063218 5607 generic.go:356] "Generic (PLEG): container finished" podID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerID="d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b" exitCode=0 Oct 30 22:49:53.063356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.063283 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" Oct 30 22:49:53.063356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.063299 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" event={"ID":"36ca4a4f-814f-4467-850f-7c72ca21d55c","Type":"ContainerDied","Data":"d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b"} Oct 30 22:49:53.063356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.063352 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8" event={"ID":"36ca4a4f-814f-4467-850f-7c72ca21d55c","Type":"ContainerDied","Data":"cc8f51d5e27d4d5a438d2da26247d3678a4156ed71cba35c249f02b419400b05"} Oct 30 22:49:53.074411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.074395 5607 scope.go:122] "RemoveContainer" containerID="39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7" Oct 30 22:49:53.074698 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:53.074676 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7\": container with ID starting with 39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7 not found: ID does not exist" containerID="39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7" Oct 30 22:49:53.074805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.074701 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7"} err="failed to get container status \"39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7\": rpc error: code = NotFound desc = could not find container \"39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7\": container with ID starting with 39f84efb9d60ef7646ed8810ac65ade1a6ba29beba81709e00ac72c8924c25c7 not found: ID does not exist" Oct 30 22:49:53.074805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.074720 5607 scope.go:122] "RemoveContainer" containerID="d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b" Oct 30 22:49:53.083811 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.083794 5607 scope.go:122] "RemoveContainer" containerID="d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b" Oct 30 22:49:53.084135 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:49:53.084111 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b\": container with ID starting with d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b not found: ID does not exist" containerID="d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b" Oct 30 22:49:53.084219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.084142 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b"} err="failed to get container status \"d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b\": rpc error: code = NotFound desc = could not find container \"d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b\": container with ID starting with d5643554b0b53fa3ae01ac1ef6bc510eacdc08b61d9304afa0b9ad87daff166b not found: ID does not exist" Oct 30 22:49:53.234594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.234562 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8"] Oct 30 22:49:53.244301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.244281 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e65019-789f4df9d5-l2fz8"] Oct 30 22:49:53.357328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.357253 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb"] Oct 30 22:49:53.359644 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:53.359624 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e65019-7b9975cdc9-ckrwb"] Oct 30 22:49:54.871593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:54.871564 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" path="/var/lib/kubelet/pods/36ca4a4f-814f-4467-850f-7c72ca21d55c/volumes" Oct 30 22:49:54.871994 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:49:54.871937 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" path="/var/lib/kubelet/pods/92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f/volumes" Oct 30 22:50:22.868129 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:50:22.868091 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:50:50.871776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.871734 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm"] Oct 30 22:50:50.872614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872591 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" containerName="payload-processing" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872615 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" containerName="payload-processing" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872632 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bbac6610-8829-4d6e-81c8-994755287db3" containerName="maas-api" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872639 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbac6610-8829-4d6e-81c8-994755287db3" containerName="maas-api" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872661 5607 cpu_manager.go:414] "containerMap: removing container" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerName="payload-pre-processing" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872668 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerName="payload-pre-processing" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872690 5607 cpu_manager.go:414] "containerMap: removing container" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872696 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872709 5607 cpu_manager.go:414] "containerMap: removing container" podUID="48bf32a3-efed-4c9a-bbd1-64815f1b60ec" containerName="payload-processing" Oct 30 22:50:50.872724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872715 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bf32a3-efed-4c9a-bbd1-64815f1b60ec" containerName="payload-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872728 5607 cpu_manager.go:414] "containerMap: removing container" podUID="805fd112-9e46-4ac0-b482-5aab9c6ec1b8" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872735 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805fd112-9e46-4ac0-b482-5aab9c6ec1b8" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872748 5607 cpu_manager.go:414] "containerMap: removing container" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerName="payload-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872755 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerName="payload-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872779 5607 cpu_manager.go:414] "containerMap: removing container" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872786 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872808 5607 cpu_manager.go:414] "containerMap: removing container" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872815 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872826 5607 cpu_manager.go:414] "containerMap: removing container" podUID="ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.872833 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873012 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bf32a3-efed-4c9a-bbd1-64815f1b60ec" containerName="payload-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873029 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="805fd112-9e46-4ac0-b482-5aab9c6ec1b8" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873040 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ca4a4f-814f-4467-850f-7c72ca21d55c" containerName="payload-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873054 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e967b0-69dd-46e9-99e5-796a073a73d8" containerName="istio-proxy" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873067 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f8222d-6aac-4906-aba3-54f1ecb79165" containerName="main" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873077 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5ca4b4-ddf1-4f24-ae00-4b3ee9bea5fc" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873092 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3e1a0b-7d10-458a-85dc-22f0fb896b9d" containerName="payload-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873101 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbac6610-8829-4d6e-81c8-994755287db3" containerName="maas-api" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873113 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fe4eec-a061-47ae-8ac5-dc9b7c4cb94f" containerName="payload-pre-processing" Oct 30 22:50:50.873250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.873125 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f3e9be-0d98-419d-8bb0-8309858d897a" containerName="main" Oct 30 22:50:50.876628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.876608 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:50.981781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.981745 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9626a6-serving-cert\"" Oct 30 22:50:50.984614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:50.984590 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm"] Oct 30 22:50:51.033433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.033405 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.033588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.033530 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7xp\" (UniqueName: \"kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.033632 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.033588 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.033787 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.033771 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.135101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.135015 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.135101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.135075 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.135336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.135152 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7xp\" (UniqueName: \"kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.135336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.135213 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.135336 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:50:51.135245 5607 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-serving-cert: secret "maas-api-e2e-extmod-a-9626a6-serving-cert" not found Oct 30 22:50:51.135336 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:50:51.135305 5607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls podName:6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d nodeName:}" failed. No retries permitted until 2026-10-30 22:50:51.635287118 +0000 UTC m=+3831.035843218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls") pod "maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" (UID: "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d") : secret "maas-api-e2e-extmod-a-9626a6-serving-cert" not found Oct 30 22:50:51.135871 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.135846 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.137735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.137711 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.198445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.198418 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7xp\" (UniqueName: \"kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.641892 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.641860 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.644439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.644412 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:51.789837 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:51.789807 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:52.164605 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:50:52.164567 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb8a7ea_b6b9_4aaa_b0af_aaa2960ba88d.slice/crio-f6e2c3d06889b9dfbed5e9048dfcd3e6dff125c9149903a6b3ce0a4e8d96cac5 WatchSource:0}: Error finding container f6e2c3d06889b9dfbed5e9048dfcd3e6dff125c9149903a6b3ce0a4e8d96cac5: Status 404 returned error can't find the container with id f6e2c3d06889b9dfbed5e9048dfcd3e6dff125c9149903a6b3ce0a4e8d96cac5 Oct 30 22:50:52.166058 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:52.166035 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:50:52.168288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:52.168260 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm"] Oct 30 22:50:52.433847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:52.433766 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" event={"ID":"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d","Type":"ContainerStarted","Data":"f6e2c3d06889b9dfbed5e9048dfcd3e6dff125c9149903a6b3ce0a4e8d96cac5"} Oct 30 22:50:52.869378 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:50:52.869263 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:50:53.441459 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:53.441424 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" event={"ID":"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d","Type":"ContainerStarted","Data":"00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022"} Oct 30 22:50:53.441873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:53.441615 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:50:53.564062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:53.564005 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" podStartSLOduration=2.596950913 podStartE2EDuration="3.56399022s" totalImagesPullingTime="967.039307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:53.562254092 +0000 UTC m=+3832.962810225" watchObservedRunningTime="2026-10-30 22:50:53.56399022 +0000 UTC m=+3832.964546496" Oct 30 22:50:59.453772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:50:59.453743 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:51:22.864152 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:51:22.864110 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:51:24.439818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:24.439786 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8"] Oct 30 22:51:24.562105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:24.562068 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n"] Oct 30 22:51:29.440937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.440888 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" podUID="37711f92-7c68-4c03-97c8-c86d2e012007" containerName="payload-processing" containerID="cri-o://0511d61a1769d76bc2dc35e9d87dec7053dc9f5e8e3f648add6a348f4dadbc6d" gracePeriod=25 Oct 30 22:51:29.562405 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.562356 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" podUID="8e411daa-bada-4599-9c96-38e1dc7d1bac" containerName="payload-pre-processing" containerID="cri-o://b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd" gracePeriod=25 Oct 30 22:51:29.679622 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.679584 5607 generic.go:356] "Generic (PLEG): container finished" podID="37711f92-7c68-4c03-97c8-c86d2e012007" containerID="0511d61a1769d76bc2dc35e9d87dec7053dc9f5e8e3f648add6a348f4dadbc6d" exitCode=0 Oct 30 22:51:29.679830 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.679677 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" event={"ID":"37711f92-7c68-4c03-97c8-c86d2e012007","Type":"ContainerDied","Data":"0511d61a1769d76bc2dc35e9d87dec7053dc9f5e8e3f648add6a348f4dadbc6d"} Oct 30 22:51:29.719619 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.719590 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:51:29.787121 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.787085 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl\" (UniqueName: \"kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl\") pod \"37711f92-7c68-4c03-97c8-c86d2e012007\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " Oct 30 22:51:29.787330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.787204 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume\") pod \"37711f92-7c68-4c03-97c8-c86d2e012007\" (UID: \"37711f92-7c68-4c03-97c8-c86d2e012007\") " Oct 30 22:51:29.787682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.787653 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume" pod "37711f92-7c68-4c03-97c8-c86d2e012007" (UID: "37711f92-7c68-4c03-97c8-c86d2e012007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:29.790531 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.790490 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl" pod "37711f92-7c68-4c03-97c8-c86d2e012007" (UID: "37711f92-7c68-4c03-97c8-c86d2e012007"). InnerVolumeSpecName "kube-api-access-zfkpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:29.815353 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.815331 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:51:29.888808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.888771 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume\") pod \"8e411daa-bada-4599-9c96-38e1dc7d1bac\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " Oct 30 22:51:29.889047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.888824 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh\" (UniqueName: \"kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh\") pod \"8e411daa-bada-4599-9c96-38e1dc7d1bac\" (UID: \"8e411daa-bada-4599-9c96-38e1dc7d1bac\") " Oct 30 22:51:29.889186 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.889164 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume" pod "8e411daa-bada-4599-9c96-38e1dc7d1bac" (UID: "8e411daa-bada-4599-9c96-38e1dc7d1bac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:29.889381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.889361 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e411daa-bada-4599-9c96-38e1dc7d1bac-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:29.889445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.889387 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkpl\" (UniqueName: \"kubernetes.io/projected/37711f92-7c68-4c03-97c8-c86d2e012007-kube-api-access-zfkpl\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:29.889445 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.889402 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37711f92-7c68-4c03-97c8-c86d2e012007-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:29.891100 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.891081 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh" pod "8e411daa-bada-4599-9c96-38e1dc7d1bac" (UID: "8e411daa-bada-4599-9c96-38e1dc7d1bac"). InnerVolumeSpecName "kube-api-access-mhjjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:29.990217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:29.990140 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjjh\" (UniqueName: \"kubernetes.io/projected/8e411daa-bada-4599-9c96-38e1dc7d1bac-kube-api-access-mhjjh\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:30.687658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.687627 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" Oct 30 22:51:30.687658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.687652 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8" event={"ID":"37711f92-7c68-4c03-97c8-c86d2e012007","Type":"ContainerDied","Data":"1ddd2aa965465a839fecc8b547b17e2e1890ee2821033e96be63c63b9e78de7f"} Oct 30 22:51:30.688174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.687717 5607 scope.go:122] "RemoveContainer" containerID="0511d61a1769d76bc2dc35e9d87dec7053dc9f5e8e3f648add6a348f4dadbc6d" Oct 30 22:51:30.689406 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.689380 5607 generic.go:356] "Generic (PLEG): container finished" podID="8e411daa-bada-4599-9c96-38e1dc7d1bac" containerID="b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd" exitCode=0 Oct 30 22:51:30.689561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.689424 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" event={"ID":"8e411daa-bada-4599-9c96-38e1dc7d1bac","Type":"ContainerDied","Data":"b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd"} Oct 30 22:51:30.689561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.689428 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" Oct 30 22:51:30.689561 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.689459 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n" event={"ID":"8e411daa-bada-4599-9c96-38e1dc7d1bac","Type":"ContainerDied","Data":"93b4bd3d5a6340f82b0030cce0942dfdbecad30be5ff3da41b2f03fba41dcb6c"} Oct 30 22:51:30.698878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.698851 5607 scope.go:122] "RemoveContainer" containerID="b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd" Oct 30 22:51:30.708705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.708686 5607 scope.go:122] "RemoveContainer" containerID="b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd" Oct 30 22:51:30.709016 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:51:30.708995 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd\": container with ID starting with b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd not found: ID does not exist" containerID="b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd" Oct 30 22:51:30.709072 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.709025 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd"} err="failed to get container status \"b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd\": rpc error: code = NotFound desc = could not find container \"b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd\": container with ID starting with b18f9989362b0d79e78dabcaa026b1ba1389e1041f617ca784558641ce6c3dbd not found: ID does not exist" Oct 30 22:51:30.856500 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.856464 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8"] Oct 30 22:51:30.907831 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.907800 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-293c80-76cd699cf-bl6t8"] Oct 30 22:51:30.980828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.980733 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n"] Oct 30 22:51:30.981114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:30.981096 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-293c80-7947b7869f-fnm9n"] Oct 30 22:51:32.871649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:32.871616 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37711f92-7c68-4c03-97c8-c86d2e012007" path="/var/lib/kubelet/pods/37711f92-7c68-4c03-97c8-c86d2e012007/volumes" Oct 30 22:51:32.872124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:32.872108 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e411daa-bada-4599-9c96-38e1dc7d1bac" path="/var/lib/kubelet/pods/8e411daa-bada-4599-9c96-38e1dc7d1bac/volumes" Oct 30 22:51:52.864082 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:51:52.864043 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:51:56.189844 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:51:56.189811 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst"] Oct 30 22:52:01.190414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:01.190357 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerName="payload-pre-processing" containerID="cri-o://342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861" gracePeriod=25 Oct 30 22:52:01.455363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:01.455294 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:52:01.463257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:01.463236 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:52:08.061629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.061596 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh"] Oct 30 22:52:08.062079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.061774 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" podUID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" containerName="maas-api" containerID="cri-o://0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9" gracePeriod=30 Oct 30 22:52:08.225259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.225212 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" podUID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.213:8443/health\": dial tcp 10.133.0.213:8443: connect: connection refused" Oct 30 22:52:08.327983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.327897 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:52:08.452482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.452447 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca" pod "50e54bdf-abbb-4a76-ae6e-c98d237359d1" (UID: "50e54bdf-abbb-4a76-ae6e-c98d237359d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:08.452723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.451608 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca\") pod \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " Oct 30 22:52:08.453037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.452781 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s\" (UniqueName: \"kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s\") pod \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " Oct 30 22:52:08.453376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.453273 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs\") pod \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " Oct 30 22:52:08.453376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.453329 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\") pod \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\" (UID: \"50e54bdf-abbb-4a76-ae6e-c98d237359d1\") " Oct 30 22:52:08.454075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.453889 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e54bdf-abbb-4a76-ae6e-c98d237359d1-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:08.456451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.456426 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs" pod "50e54bdf-abbb-4a76-ae6e-c98d237359d1" (UID: "50e54bdf-abbb-4a76-ae6e-c98d237359d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:08.456451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.456618 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s" pod "50e54bdf-abbb-4a76-ae6e-c98d237359d1" (UID: "50e54bdf-abbb-4a76-ae6e-c98d237359d1"). InnerVolumeSpecName "kube-api-access-vtl6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:08.466598 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.463596 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls" pod "50e54bdf-abbb-4a76-ae6e-c98d237359d1" (UID: "50e54bdf-abbb-4a76-ae6e-c98d237359d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:08.554888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.554856 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtl6s\" (UniqueName: \"kubernetes.io/projected/50e54bdf-abbb-4a76-ae6e-c98d237359d1-kube-api-access-vtl6s\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:08.554888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.554884 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:08.554888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.554896 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e54bdf-abbb-4a76-ae6e-c98d237359d1-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:08.925222 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.925195 5607 generic.go:356] "Generic (PLEG): container finished" podID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" containerID="0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9" exitCode=0 Oct 30 22:52:08.925373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.925248 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" Oct 30 22:52:08.925373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.925257 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" event={"ID":"50e54bdf-abbb-4a76-ae6e-c98d237359d1","Type":"ContainerDied","Data":"0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9"} Oct 30 22:52:08.925373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.925295 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh" event={"ID":"50e54bdf-abbb-4a76-ae6e-c98d237359d1","Type":"ContainerDied","Data":"72ae8a80822eaf7181bdf1e6f5992790982d0ac0cb6b5ae13cbc35413601cf7e"} Oct 30 22:52:08.925373 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.925317 5607 scope.go:122] "RemoveContainer" containerID="0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9" Oct 30 22:52:08.937232 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.937210 5607 scope.go:122] "RemoveContainer" containerID="0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9" Oct 30 22:52:08.937526 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:08.937503 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9\": container with ID starting with 0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9 not found: ID does not exist" containerID="0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9" Oct 30 22:52:08.937586 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:08.937534 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9"} err="failed to get container status \"0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9\": rpc error: code = NotFound desc = could not find container \"0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9\": container with ID starting with 0fb3bfb5e54d922db7b1f1aa03d336448b92a2c65444412cb70cbb13ee583de9 not found: ID does not exist" Oct 30 22:52:09.094902 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:09.094871 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh"] Oct 30 22:52:09.099719 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:09.099688 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f39eb7-7b67fdfb48-pd2sh"] Oct 30 22:52:10.641344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:10.641301 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:10.872304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:10.872272 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" path="/var/lib/kubelet/pods/50e54bdf-abbb-4a76-ae6e-c98d237359d1/volumes" Oct 30 22:52:20.641740 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:20.641704 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:21.455447 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.455388 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:52:21.500746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.500720 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb\" (UniqueName: \"kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb\") pod \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " Oct 30 22:52:21.500889 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.500778 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume\") pod \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\" (UID: \"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db\") " Oct 30 22:52:21.501195 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.501169 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume" pod "85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" (UID: "85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:21.501362 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.501339 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:21.503174 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.503150 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb" pod "85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" (UID: "85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db"). InnerVolumeSpecName "kube-api-access-frczb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:21.603144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:21.603114 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frczb\" (UniqueName: \"kubernetes.io/projected/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db-kube-api-access-frczb\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:22.024249 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.024209 5607 generic.go:356] "Generic (PLEG): container finished" podID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerID="342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861" exitCode=0 Oct 30 22:52:22.024745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.024260 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" Oct 30 22:52:22.024745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.024293 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" event={"ID":"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db","Type":"ContainerDied","Data":"342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861"} Oct 30 22:52:22.024745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.024348 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst" event={"ID":"85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db","Type":"ContainerDied","Data":"cbac28d8ec491dc9a5ff7f41b9f80dc65fb63c2bd503f87999c3a72167f4a324"} Oct 30 22:52:22.024745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.024373 5607 scope.go:122] "RemoveContainer" containerID="342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861" Oct 30 22:52:22.037480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.037435 5607 scope.go:122] "RemoveContainer" containerID="342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861" Oct 30 22:52:22.037771 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:22.037753 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861\": container with ID starting with 342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861 not found: ID does not exist" containerID="342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861" Oct 30 22:52:22.037847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.037782 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861"} err="failed to get container status \"342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861\": rpc error: code = NotFound desc = could not find container \"342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861\": container with ID starting with 342b8b4a7a93b25b635814c3bf3acbd5e10dc7bc06700e41d171214c9fae9861 not found: ID does not exist" Oct 30 22:52:22.196491 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.196460 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst"] Oct 30 22:52:22.247339 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.247314 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f39eb7-64dbc9bd59-vzxst"] Oct 30 22:52:22.862854 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:22.862818 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:52:22.871239 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:22.871211 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" path="/var/lib/kubelet/pods/85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db/volumes" Oct 30 22:52:29.876544 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:29.876511 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r"] Oct 30 22:52:29.988262 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:29.988231 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5"] Oct 30 22:52:34.877468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:34.877433 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerName="payload-processing" containerID="cri-o://e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487" gracePeriod=25 Oct 30 22:52:34.989009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:34.988975 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerName="payload-pre-processing" containerID="cri-o://3f2a8a8186adfd44534679e5c45733e14d7c49b9ff56724c21f95a2248af21f6" gracePeriod=25 Oct 30 22:52:38.970406 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:38.970365 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6"] Oct 30 22:52:41.122785 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.122750 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z"] Oct 30 22:52:41.123234 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.122964 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" podUID="a4c5d315-965e-4315-a243-0af5c701cfd2" containerName="maas-api" containerID="cri-o://0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498" gracePeriod=30 Oct 30 22:52:41.382757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.382703 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:52:41.518687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.518632 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs\") pod \"a4c5d315-965e-4315-a243-0af5c701cfd2\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " Oct 30 22:52:41.518885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.518701 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj\" (UniqueName: \"kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj\") pod \"a4c5d315-965e-4315-a243-0af5c701cfd2\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " Oct 30 22:52:41.518885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.518741 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca\") pod \"a4c5d315-965e-4315-a243-0af5c701cfd2\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " Oct 30 22:52:41.518885 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.518869 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\") pod \"a4c5d315-965e-4315-a243-0af5c701cfd2\" (UID: \"a4c5d315-965e-4315-a243-0af5c701cfd2\") " Oct 30 22:52:41.519973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.519924 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca" pod "a4c5d315-965e-4315-a243-0af5c701cfd2" (UID: "a4c5d315-965e-4315-a243-0af5c701cfd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:41.521706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.521677 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls" pod "a4c5d315-965e-4315-a243-0af5c701cfd2" (UID: "a4c5d315-965e-4315-a243-0af5c701cfd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:41.522797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.522771 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj" pod "a4c5d315-965e-4315-a243-0af5c701cfd2" (UID: "a4c5d315-965e-4315-a243-0af5c701cfd2"). InnerVolumeSpecName "kube-api-access-w8jfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:41.526075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.526045 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs" pod "a4c5d315-965e-4315-a243-0af5c701cfd2" (UID: "a4c5d315-965e-4315-a243-0af5c701cfd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:41.620594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.620493 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:41.620594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.620541 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4c5d315-965e-4315-a243-0af5c701cfd2-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:41.620594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.620558 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jfj\" (UniqueName: \"kubernetes.io/projected/a4c5d315-965e-4315-a243-0af5c701cfd2-kube-api-access-w8jfj\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:41.620594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:41.620571 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5d315-965e-4315-a243-0af5c701cfd2-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:42.161593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.161558 5607 generic.go:356] "Generic (PLEG): container finished" podID="a4c5d315-965e-4315-a243-0af5c701cfd2" containerID="0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498" exitCode=0 Oct 30 22:52:42.162093 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.161602 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" Oct 30 22:52:42.162093 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.161605 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" event={"ID":"a4c5d315-965e-4315-a243-0af5c701cfd2","Type":"ContainerDied","Data":"0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498"} Oct 30 22:52:42.162093 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.161662 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z" event={"ID":"a4c5d315-965e-4315-a243-0af5c701cfd2","Type":"ContainerDied","Data":"32c8fc4945865985ad7ddf807ab54c1fc39501733bc52b9775c90a75ea8a806a"} Oct 30 22:52:42.162093 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.161683 5607 scope.go:122] "RemoveContainer" containerID="0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498" Oct 30 22:52:42.172828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.172813 5607 scope.go:122] "RemoveContainer" containerID="0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498" Oct 30 22:52:42.173115 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:42.173098 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498\": container with ID starting with 0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498 not found: ID does not exist" containerID="0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498" Oct 30 22:52:42.173165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.173123 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498"} err="failed to get container status \"0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498\": rpc error: code = NotFound desc = could not find container \"0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498\": container with ID starting with 0db7c6a4d2c6edbaf51b9e4f155801e3601e9c27b3f8cd14585dfe2056688498 not found: ID does not exist" Oct 30 22:52:42.330070 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.330043 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z"] Oct 30 22:52:42.380927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.380905 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-37e79b-78697bc7cd-vgc2z"] Oct 30 22:52:42.785391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.785351 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:42.870677 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:42.870646 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c5d315-965e-4315-a243-0af5c701cfd2" path="/var/lib/kubelet/pods/a4c5d315-965e-4315-a243-0af5c701cfd2/volumes" Oct 30 22:52:43.971450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:43.971409 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" podUID="3c486f28-0c4c-4065-910a-3d590c82003d" containerName="payload-pre-processing" containerID="cri-o://30bf3c9968640ed90e54e169e5f1c4c7ff4c2a4547ce3bd21b22934cb89361f3" gracePeriod=25 Oct 30 22:52:44.177338 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.177306 5607 generic.go:356] "Generic (PLEG): container finished" podID="3c486f28-0c4c-4065-910a-3d590c82003d" containerID="30bf3c9968640ed90e54e169e5f1c4c7ff4c2a4547ce3bd21b22934cb89361f3" exitCode=0 Oct 30 22:52:44.177507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.177386 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" event={"ID":"3c486f28-0c4c-4065-910a-3d590c82003d","Type":"ContainerDied","Data":"30bf3c9968640ed90e54e169e5f1c4c7ff4c2a4547ce3bd21b22934cb89361f3"} Oct 30 22:52:44.258039 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.258013 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:52:44.351554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.351526 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft\" (UniqueName: \"kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft\") pod \"3c486f28-0c4c-4065-910a-3d590c82003d\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " Oct 30 22:52:44.351727 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.351600 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume\") pod \"3c486f28-0c4c-4065-910a-3d590c82003d\" (UID: \"3c486f28-0c4c-4065-910a-3d590c82003d\") " Oct 30 22:52:44.352079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.352055 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume" pod "3c486f28-0c4c-4065-910a-3d590c82003d" (UID: "3c486f28-0c4c-4065-910a-3d590c82003d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:44.354026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.353996 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft" pod "3c486f28-0c4c-4065-910a-3d590c82003d" (UID: "3c486f28-0c4c-4065-910a-3d590c82003d"). InnerVolumeSpecName "kube-api-access-p46ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:44.453520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.453494 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p46ft\" (UniqueName: \"kubernetes.io/projected/3c486f28-0c4c-4065-910a-3d590c82003d-kube-api-access-p46ft\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:44.453520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.453519 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c486f28-0c4c-4065-910a-3d590c82003d-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:44.797911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:44.797879 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:45.184800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:45.184768 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" event={"ID":"3c486f28-0c4c-4065-910a-3d590c82003d","Type":"ContainerDied","Data":"a44efa2f2317ec4933dcc50236b8d2b8c81593bb636ee01292552f2af35864d3"} Oct 30 22:52:45.185313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:45.184843 5607 scope.go:122] "RemoveContainer" containerID="30bf3c9968640ed90e54e169e5f1c4c7ff4c2a4547ce3bd21b22934cb89361f3" Oct 30 22:52:45.185313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:45.184786 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6" Oct 30 22:52:45.355107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:45.355078 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6"] Oct 30 22:52:45.405757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:45.405731 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-421286-db4c74cb7-ph9l6"] Oct 30 22:52:46.870468 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:46.870435 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c486f28-0c4c-4065-910a-3d590c82003d" path="/var/lib/kubelet/pods/3c486f28-0c4c-4065-910a-3d590c82003d/volumes" Oct 30 22:52:52.785451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:52.785412 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:52.864674 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:52.864634 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:52:54.798474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:54.798430 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:55.172286 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.172165 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:52:55.253674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.253648 5607 generic.go:356] "Generic (PLEG): container finished" podID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerID="e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487" exitCode=0 Oct 30 22:52:55.253835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.253697 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" Oct 30 22:52:55.253835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.253725 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" event={"ID":"810d67a6-711d-44c7-a72f-5b6e1b405026","Type":"ContainerDied","Data":"e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487"} Oct 30 22:52:55.253835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.253789 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r" event={"ID":"810d67a6-711d-44c7-a72f-5b6e1b405026","Type":"ContainerDied","Data":"1538067fbeecff1a67ab1e7af602482621fb5ab38fed49a0c7f3ee616f0b7735"} Oct 30 22:52:55.253835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.253818 5607 scope.go:122] "RemoveContainer" containerID="e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487" Oct 30 22:52:55.255006 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.254984 5607 generic.go:356] "Generic (PLEG): container finished" podID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerID="3f2a8a8186adfd44534679e5c45733e14d7c49b9ff56724c21f95a2248af21f6" exitCode=0 Oct 30 22:52:55.255107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.255034 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" event={"ID":"a59dd0a1-3355-4607-aecc-d959da666b1a","Type":"ContainerDied","Data":"3f2a8a8186adfd44534679e5c45733e14d7c49b9ff56724c21f95a2248af21f6"} Oct 30 22:52:55.255107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.255068 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" event={"ID":"a59dd0a1-3355-4607-aecc-d959da666b1a","Type":"ContainerDied","Data":"9504dd9120d069b09b621705bad08334484806d8dae19162c98fe2a97c2dba4d"} Oct 30 22:52:55.255107 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.255083 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9504dd9120d069b09b621705bad08334484806d8dae19162c98fe2a97c2dba4d" Oct 30 22:52:55.256115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.256093 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:52:55.263480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.263466 5607 scope.go:122] "RemoveContainer" containerID="e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487" Oct 30 22:52:55.263747 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:55.263728 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487\": container with ID starting with e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487 not found: ID does not exist" containerID="e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487" Oct 30 22:52:55.263820 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.263755 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487"} err="failed to get container status \"e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487\": rpc error: code = NotFound desc = could not find container \"e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487\": container with ID starting with e123e99a1dfc89e5e2b66f59121d896acb1a714ec5d33fc6f21e89a57f84e487 not found: ID does not exist" Oct 30 22:52:55.271206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.271189 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume\") pod \"810d67a6-711d-44c7-a72f-5b6e1b405026\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " Oct 30 22:52:55.271298 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.271286 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg\" (UniqueName: \"kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg\") pod \"810d67a6-711d-44c7-a72f-5b6e1b405026\" (UID: \"810d67a6-711d-44c7-a72f-5b6e1b405026\") " Oct 30 22:52:55.271521 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.271505 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume" pod "810d67a6-711d-44c7-a72f-5b6e1b405026" (UID: "810d67a6-711d-44c7-a72f-5b6e1b405026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:55.273315 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.273292 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg" pod "810d67a6-711d-44c7-a72f-5b6e1b405026" (UID: "810d67a6-711d-44c7-a72f-5b6e1b405026"). InnerVolumeSpecName "kube-api-access-w52tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:55.372335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.372259 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume\") pod \"a59dd0a1-3355-4607-aecc-d959da666b1a\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " Oct 30 22:52:55.372335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.372302 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5\" (UniqueName: \"kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5\") pod \"a59dd0a1-3355-4607-aecc-d959da666b1a\" (UID: \"a59dd0a1-3355-4607-aecc-d959da666b1a\") " Oct 30 22:52:55.372633 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.372611 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume" pod "a59dd0a1-3355-4607-aecc-d959da666b1a" (UID: "a59dd0a1-3355-4607-aecc-d959da666b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:55.372751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.372739 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/810d67a6-711d-44c7-a72f-5b6e1b405026-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.372800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.372753 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59dd0a1-3355-4607-aecc-d959da666b1a-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.372800 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.372763 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w52tg\" (UniqueName: \"kubernetes.io/projected/810d67a6-711d-44c7-a72f-5b6e1b405026-kube-api-access-w52tg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.374425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.374404 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5" pod "a59dd0a1-3355-4607-aecc-d959da666b1a" (UID: "a59dd0a1-3355-4607-aecc-d959da666b1a"). InnerVolumeSpecName "kube-api-access-4z6j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:55.474101 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.474074 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6j5\" (UniqueName: \"kubernetes.io/projected/a59dd0a1-3355-4607-aecc-d959da666b1a-kube-api-access-4z6j5\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:55.677448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.677380 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r"] Oct 30 22:52:55.682913 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.682888 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-37e79b-7f869b84f-tgw8r"] Oct 30 22:52:55.707136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.707109 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx"] Oct 30 22:52:55.712098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.712081 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg"] Oct 30 22:52:55.732726 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:55.732702 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h"] Oct 30 22:52:56.263167 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.263137 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5" Oct 30 22:52:56.434162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.434130 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5"] Oct 30 22:52:56.439027 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.439002 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-37e79b-69f97b7455-7cln5"] Oct 30 22:52:56.810858 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.810825 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd"] Oct 30 22:52:56.811133 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.811108 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" podUID="69f559a6-a076-4147-8069-694ec06a8575" containerName="maas-api" containerID="cri-o://f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1" gracePeriod=30 Oct 30 22:52:56.819511 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.819487 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh"] Oct 30 22:52:56.871530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.871504 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" path="/var/lib/kubelet/pods/810d67a6-711d-44c7-a72f-5b6e1b405026/volumes" Oct 30 22:52:56.871978 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:56.871944 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" path="/var/lib/kubelet/pods/a59dd0a1-3355-4607-aecc-d959da666b1a/volumes" Oct 30 22:52:57.100024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.100001 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:52:57.189085 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.189058 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs\") pod \"69f559a6-a076-4147-8069-694ec06a8575\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " Oct 30 22:52:57.189273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.189127 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\") pod \"69f559a6-a076-4147-8069-694ec06a8575\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " Oct 30 22:52:57.189273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.189229 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca\") pod \"69f559a6-a076-4147-8069-694ec06a8575\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " Oct 30 22:52:57.189437 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.189422 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr\" (UniqueName: \"kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr\") pod \"69f559a6-a076-4147-8069-694ec06a8575\" (UID: \"69f559a6-a076-4147-8069-694ec06a8575\") " Oct 30 22:52:57.189597 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.189569 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca" pod "69f559a6-a076-4147-8069-694ec06a8575" (UID: "69f559a6-a076-4147-8069-694ec06a8575"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:57.189927 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.189906 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f559a6-a076-4147-8069-694ec06a8575-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:57.191334 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.191314 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls" pod "69f559a6-a076-4147-8069-694ec06a8575" (UID: "69f559a6-a076-4147-8069-694ec06a8575"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:57.191998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.191943 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs" pod "69f559a6-a076-4147-8069-694ec06a8575" (UID: "69f559a6-a076-4147-8069-694ec06a8575"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:57.192143 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.192004 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr" pod "69f559a6-a076-4147-8069-694ec06a8575" (UID: "69f559a6-a076-4147-8069-694ec06a8575"). InnerVolumeSpecName "kube-api-access-r9qbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:57.274196 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.274162 5607 generic.go:356] "Generic (PLEG): container finished" podID="69f559a6-a076-4147-8069-694ec06a8575" containerID="f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1" exitCode=0 Oct 30 22:52:57.274686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.274225 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" Oct 30 22:52:57.274686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.274256 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" event={"ID":"69f559a6-a076-4147-8069-694ec06a8575","Type":"ContainerDied","Data":"f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1"} Oct 30 22:52:57.274686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.274308 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd" event={"ID":"69f559a6-a076-4147-8069-694ec06a8575","Type":"ContainerDied","Data":"df228c57fb3377b8bcf73d5ca58a903f888956010a1e32d6cd73195e159b9ef8"} Oct 30 22:52:57.274686 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.274334 5607 scope.go:122] "RemoveContainer" containerID="f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1" Oct 30 22:52:57.288104 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.288051 5607 scope.go:122] "RemoveContainer" containerID="f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1" Oct 30 22:52:57.288364 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:52:57.288333 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1\": container with ID starting with f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1 not found: ID does not exist" containerID="f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1" Oct 30 22:52:57.288431 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.288378 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1"} err="failed to get container status \"f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1\": rpc error: code = NotFound desc = could not find container \"f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1\": container with ID starting with f534bf3a16ffb7538603fb6ca8a52337c11690bbbf047431150dd1835a98a6a1 not found: ID does not exist" Oct 30 22:52:57.292021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.292000 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9qbr\" (UniqueName: \"kubernetes.io/projected/69f559a6-a076-4147-8069-694ec06a8575-kube-api-access-r9qbr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:57.292021 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.292022 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:57.292146 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.292031 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f559a6-a076-4147-8069-694ec06a8575-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:57.444721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.444686 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd"] Oct 30 22:52:57.450425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:57.450397 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc6466-d844b4dfb-wlwkd"] Oct 30 22:52:58.870329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:52:58.870290 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f559a6-a076-4147-8069-694ec06a8575" path="/var/lib/kubelet/pods/69f559a6-a076-4147-8069-694ec06a8575/volumes" Oct 30 22:53:01.820464 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:01.820421 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerName="payload-processing" containerID="cri-o://144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168" gracePeriod=25 Oct 30 22:53:02.736604 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:02.736573 5607 scope.go:122] "RemoveContainer" containerID="3f2a8a8186adfd44534679e5c45733e14d7c49b9ff56724c21f95a2248af21f6" Oct 30 22:53:03.593162 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:03.593129 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548"] Oct 30 22:53:08.593923 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:08.593880 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" podUID="375990b8-fabe-4822-881d-d776174dd6ec" containerName="payload-pre-processing" containerID="cri-o://c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5" gracePeriod=25 Oct 30 22:53:10.091001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:10.090946 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:10.729042 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:10.729001 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" containerID="cri-o://86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478" gracePeriod=45 Oct 30 22:53:10.734157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:10.734091 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" containerID="cri-o://b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c" gracePeriod=45 Oct 30 22:53:10.754305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:10.754255 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" containerID="cri-o://8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198" gracePeriod=45 Oct 30 22:53:10.789318 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:10.789287 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused" Oct 30 22:53:10.933354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:10.930322 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": dial tcp 10.133.0.84:8000: connect: connection refused" Oct 30 22:53:11.048197 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.048165 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:53:11.056002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.055980 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:53:11.060361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.060337 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:53:11.139982 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.139918 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache\") pod \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.139997 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home\") pod \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140047 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir\") pod \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140071 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs\") pod \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140097 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir\") pod \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140124 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm\") pod \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140163 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm\") pod \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140211 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache\") pod \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140261 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg\" (UniqueName: \"kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg\") pod \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140302 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs\") pod \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140361 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx\" (UniqueName: \"kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx\") pod \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\" (UID: \"c0a7fe83-aac9-44eb-847a-d3246cd0a1de\") " Oct 30 22:53:11.140436 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.140438 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home\") pod \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\" (UID: \"6a6e63ab-61f5-4da2-80e1-eac2983b35fe\") " Oct 30 22:53:11.141374 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.141157 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home" pod "6a6e63ab-61f5-4da2-80e1-eac2983b35fe" (UID: "6a6e63ab-61f5-4da2-80e1-eac2983b35fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.141488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.141414 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir" pod "c0a7fe83-aac9-44eb-847a-d3246cd0a1de" (UID: "c0a7fe83-aac9-44eb-847a-d3246cd0a1de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.141488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.141434 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache" pod "6a6e63ab-61f5-4da2-80e1-eac2983b35fe" (UID: "6a6e63ab-61f5-4da2-80e1-eac2983b35fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.141684 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.141663 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir" pod "6a6e63ab-61f5-4da2-80e1-eac2983b35fe" (UID: "6a6e63ab-61f5-4da2-80e1-eac2983b35fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.142104 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.141930 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home" pod "c0a7fe83-aac9-44eb-847a-d3246cd0a1de" (UID: "c0a7fe83-aac9-44eb-847a-d3246cd0a1de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.142469 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.142421 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache" pod "c0a7fe83-aac9-44eb-847a-d3246cd0a1de" (UID: "c0a7fe83-aac9-44eb-847a-d3246cd0a1de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.146311 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.146292 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs" pod "c0a7fe83-aac9-44eb-847a-d3246cd0a1de" (UID: "c0a7fe83-aac9-44eb-847a-d3246cd0a1de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:11.148312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.148264 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg" pod "6a6e63ab-61f5-4da2-80e1-eac2983b35fe" (UID: "6a6e63ab-61f5-4da2-80e1-eac2983b35fe"). InnerVolumeSpecName "kube-api-access-4qvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:11.148312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.148269 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm" pod "c0a7fe83-aac9-44eb-847a-d3246cd0a1de" (UID: "c0a7fe83-aac9-44eb-847a-d3246cd0a1de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.148312 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.148294 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx" pod "c0a7fe83-aac9-44eb-847a-d3246cd0a1de" (UID: "c0a7fe83-aac9-44eb-847a-d3246cd0a1de"). InnerVolumeSpecName "kube-api-access-8wjqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:11.148474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.148416 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs" pod "6a6e63ab-61f5-4da2-80e1-eac2983b35fe" (UID: "6a6e63ab-61f5-4da2-80e1-eac2983b35fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:11.149558 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.149536 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm" pod "6a6e63ab-61f5-4da2-80e1-eac2983b35fe" (UID: "6a6e63ab-61f5-4da2-80e1-eac2983b35fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.242287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242261 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir\") pod \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " Oct 30 22:53:11.242475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242332 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm\") pod \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " Oct 30 22:53:11.242475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242385 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd\" (UniqueName: \"kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd\") pod \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " Oct 30 22:53:11.242475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242457 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache\") pod \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " Oct 30 22:53:11.242663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242544 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs\" (UniqueName: \"kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs\") pod \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " Oct 30 22:53:11.242663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242555 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir" pod "450a2dde-cc87-40f6-bae8-ce0d32dd1428" (UID: "450a2dde-cc87-40f6-bae8-ce0d32dd1428"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.242663 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242570 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home\") pod \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\" (UID: \"450a2dde-cc87-40f6-bae8-ce0d32dd1428\") " Oct 30 22:53:11.242827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242801 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache" pod "450a2dde-cc87-40f6-bae8-ce0d32dd1428" (UID: "450a2dde-cc87-40f6-bae8-ce0d32dd1428"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.242827 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.242814 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home" pod "450a2dde-cc87-40f6-bae8-ce0d32dd1428" (UID: "450a2dde-cc87-40f6-bae8-ce0d32dd1428"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.243138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243111 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243138 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243133 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243149 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243162 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243175 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243185 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243194 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243201 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243208 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243215 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243226 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243241 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvzg\" (UniqueName: \"kubernetes.io/projected/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-kube-api-access-4qvzg\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243251 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6e63ab-61f5-4da2-80e1-eac2983b35fe-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243259 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.243467 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.243269 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjqx\" (UniqueName: \"kubernetes.io/projected/c0a7fe83-aac9-44eb-847a-d3246cd0a1de-kube-api-access-8wjqx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.244470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.244454 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm" pod "450a2dde-cc87-40f6-bae8-ce0d32dd1428" (UID: "450a2dde-cc87-40f6-bae8-ce0d32dd1428"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:11.244580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.244561 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd" pod "450a2dde-cc87-40f6-bae8-ce0d32dd1428" (UID: "450a2dde-cc87-40f6-bae8-ce0d32dd1428"). InnerVolumeSpecName "kube-api-access-nl8zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:11.244867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.244852 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs" pod "450a2dde-cc87-40f6-bae8-ce0d32dd1428" (UID: "450a2dde-cc87-40f6-bae8-ce0d32dd1428"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:11.344329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.344248 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450a2dde-cc87-40f6-bae8-ce0d32dd1428-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.344329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.344280 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8zd\" (UniqueName: \"kubernetes.io/projected/450a2dde-cc87-40f6-bae8-ce0d32dd1428-kube-api-access-nl8zd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.344329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.344295 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450a2dde-cc87-40f6-bae8-ce0d32dd1428-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:11.358687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.358656 5607 generic.go:356] "Generic (PLEG): container finished" podID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerID="8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198" exitCode=0 Oct 30 22:53:11.358910 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.358719 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" Oct 30 22:53:11.358910 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.358741 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" event={"ID":"450a2dde-cc87-40f6-bae8-ce0d32dd1428","Type":"ContainerDied","Data":"8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198"} Oct 30 22:53:11.358910 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.358789 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h" event={"ID":"450a2dde-cc87-40f6-bae8-ce0d32dd1428","Type":"ContainerDied","Data":"e16c80b8043733b24cf5156535066e4229f2277f3a8fcf92a875055375fecdd0"} Oct 30 22:53:11.358910 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.358810 5607 scope.go:122] "RemoveContainer" containerID="8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198" Oct 30 22:53:11.360325 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.360299 5607 generic.go:356] "Generic (PLEG): container finished" podID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerID="86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478" exitCode=0 Oct 30 22:53:11.360579 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.360411 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" event={"ID":"6a6e63ab-61f5-4da2-80e1-eac2983b35fe","Type":"ContainerDied","Data":"86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478"} Oct 30 22:53:11.360579 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.360434 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" event={"ID":"6a6e63ab-61f5-4da2-80e1-eac2983b35fe","Type":"ContainerDied","Data":"972a97287b47316a5ad33653dd425d043a656e06837f53811771099e0d5ae200"} Oct 30 22:53:11.360579 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.360497 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" Oct 30 22:53:11.362801 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.362773 5607 generic.go:356] "Generic (PLEG): container finished" podID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerID="b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c" exitCode=0 Oct 30 22:53:11.362942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.362854 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" event={"ID":"c0a7fe83-aac9-44eb-847a-d3246cd0a1de","Type":"ContainerDied","Data":"b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c"} Oct 30 22:53:11.362942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.362881 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" event={"ID":"c0a7fe83-aac9-44eb-847a-d3246cd0a1de","Type":"ContainerDied","Data":"2a1013ff89d97faaa29119c3864978f3b30fa0d61caefd9498ab8105c35182ae"} Oct 30 22:53:11.362942 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.362881 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg" Oct 30 22:53:11.371029 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.371013 5607 scope.go:122] "RemoveContainer" containerID="8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198" Oct 30 22:53:11.371316 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:11.371298 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198\": container with ID starting with 8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198 not found: ID does not exist" containerID="8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198" Oct 30 22:53:11.371391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.371328 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198"} err="failed to get container status \"8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198\": rpc error: code = NotFound desc = could not find container \"8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198\": container with ID starting with 8e631664ce3ca692d5229e5ebc67a53a78f727580e4505355773f29862674198 not found: ID does not exist" Oct 30 22:53:11.371391 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.371352 5607 scope.go:122] "RemoveContainer" containerID="86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478" Oct 30 22:53:11.379869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.379853 5607 scope.go:122] "RemoveContainer" containerID="86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478" Oct 30 22:53:11.380228 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:11.380209 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478\": container with ID starting with 86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478 not found: ID does not exist" containerID="86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478" Oct 30 22:53:11.380298 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.380233 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478"} err="failed to get container status \"86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478\": rpc error: code = NotFound desc = could not find container \"86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478\": container with ID starting with 86962bd920ae153a98d99ae2df9a491c7f1f80993e9953f6ba1ac744a8f1a478 not found: ID does not exist" Oct 30 22:53:11.380298 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.380248 5607 scope.go:122] "RemoveContainer" containerID="b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c" Oct 30 22:53:11.388161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.388141 5607 scope.go:122] "RemoveContainer" containerID="b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c" Oct 30 22:53:11.388411 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:11.388391 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c\": container with ID starting with b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c not found: ID does not exist" containerID="b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c" Oct 30 22:53:11.388481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.388417 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c"} err="failed to get container status \"b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c\": rpc error: code = NotFound desc = could not find container \"b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c\": container with ID starting with b04362749fe0c65be0fea35c8cb640260370c8ad75243859c14d878a1c9cd77c not found: ID does not exist" Oct 30 22:53:11.523807 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.523778 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg"] Oct 30 22:53:11.525939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.525916 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-w4-e524ce-kserve-894955b77-v5qvg"] Oct 30 22:53:11.637702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.637607 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h"] Oct 30 22:53:11.640148 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.640126 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/simulator-w4-e524ce-kserve-5b6cc56dc-65s7h"] Oct 30 22:53:11.801557 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.801533 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx"] Oct 30 22:53:11.812883 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:11.812858 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx"] Oct 30 22:53:12.044044 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:12.044004 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/distinct-2-w4-e524ce-kserve-64566dbc78-w5dkx" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:53:12.870409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:12.870380 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" path="/var/lib/kubelet/pods/450a2dde-cc87-40f6-bae8-ce0d32dd1428/volumes" Oct 30 22:53:12.870828 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:12.870813 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" path="/var/lib/kubelet/pods/6a6e63ab-61f5-4da2-80e1-eac2983b35fe/volumes" Oct 30 22:53:12.871274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:12.871258 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" path="/var/lib/kubelet/pods/c0a7fe83-aac9-44eb-847a-d3246cd0a1de/volumes" Oct 30 22:53:14.850205 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:14.850167 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" podUID="375990b8-fabe-4822-881d-d776174dd6ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:20.091116 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:20.091072 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:21.264998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:21.264967 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n"] Oct 30 22:53:21.372077 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:21.372049 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77"] Oct 30 22:53:22.083227 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.083168 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:53:22.145529 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.145507 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz\" (UniqueName: \"kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz\") pod \"30b73df7-7c61-4a96-8813-cb5b3548b31c\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " Oct 30 22:53:22.145680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.145554 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume\") pod \"30b73df7-7c61-4a96-8813-cb5b3548b31c\" (UID: \"30b73df7-7c61-4a96-8813-cb5b3548b31c\") " Oct 30 22:53:22.145974 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.145935 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume" pod "30b73df7-7c61-4a96-8813-cb5b3548b31c" (UID: "30b73df7-7c61-4a96-8813-cb5b3548b31c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:22.147825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.147798 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz" pod "30b73df7-7c61-4a96-8813-cb5b3548b31c" (UID: "30b73df7-7c61-4a96-8813-cb5b3548b31c"). InnerVolumeSpecName "kube-api-access-p6vgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:22.247305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.247275 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vgz\" (UniqueName: \"kubernetes.io/projected/30b73df7-7c61-4a96-8813-cb5b3548b31c-kube-api-access-p6vgz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.247305 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.247301 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b73df7-7c61-4a96-8813-cb5b3548b31c-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:22.431064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.430984 5607 generic.go:356] "Generic (PLEG): container finished" podID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerID="144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168" exitCode=0 Oct 30 22:53:22.431064 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.431042 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" event={"ID":"30b73df7-7c61-4a96-8813-cb5b3548b31c","Type":"ContainerDied","Data":"144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168"} Oct 30 22:53:22.431553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.431064 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" Oct 30 22:53:22.431553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.431119 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh" event={"ID":"30b73df7-7c61-4a96-8813-cb5b3548b31c","Type":"ContainerDied","Data":"3954c3cb7e6571905ffad5dc1f738f2c51a9a026c98a7dba9a8e77349cbbc2e4"} Oct 30 22:53:22.431553 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.431145 5607 scope.go:122] "RemoveContainer" containerID="144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168" Oct 30 22:53:22.448886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.448866 5607 scope.go:122] "RemoveContainer" containerID="144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168" Oct 30 22:53:22.449172 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:22.449147 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168\": container with ID starting with 144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168 not found: ID does not exist" containerID="144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168" Oct 30 22:53:22.449295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.449183 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168"} err="failed to get container status \"144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168\": rpc error: code = NotFound desc = could not find container \"144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168\": container with ID starting with 144143baa775ea93a60cef0aa072fb5777d6bdefa02dac6cfb481cb8fac0c168 not found: ID does not exist" Oct 30 22:53:22.602585 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.602557 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh"] Oct 30 22:53:22.650475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.650445 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fc6466-59f85f75d9-8jrsh"] Oct 30 22:53:22.870428 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:22.870382 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:53:22.873135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:22.873111 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" path="/var/lib/kubelet/pods/30b73df7-7c61-4a96-8813-cb5b3548b31c/volumes" Oct 30 22:53:24.849705 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:24.849664 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" podUID="375990b8-fabe-4822-881d-d776174dd6ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:26.265888 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:26.265849 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerName="payload-processing" containerID="cri-o://284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316" gracePeriod=25 Oct 30 22:53:26.422348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:26.422304 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerName="payload-pre-processing" containerID="cri-o://53de3ccb94a6a5e574606e537c8c29d423d37450376f958ee855afa1f8adfda0" gracePeriod=25 Oct 30 22:53:28.855878 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:28.855820 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:53:28.913545 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:28.913521 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q\") pod \"375990b8-fabe-4822-881d-d776174dd6ec\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " Oct 30 22:53:28.913706 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:28.913566 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume\") pod \"375990b8-fabe-4822-881d-d776174dd6ec\" (UID: \"375990b8-fabe-4822-881d-d776174dd6ec\") " Oct 30 22:53:28.914008 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:28.913982 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume" pod "375990b8-fabe-4822-881d-d776174dd6ec" (UID: "375990b8-fabe-4822-881d-d776174dd6ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:28.915721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:28.915701 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q" pod "375990b8-fabe-4822-881d-d776174dd6ec" (UID: "375990b8-fabe-4822-881d-d776174dd6ec"). InnerVolumeSpecName "kube-api-access-pgf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:29.015361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.015330 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/375990b8-fabe-4822-881d-d776174dd6ec-kube-api-access-pgf8q\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.015361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.015357 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375990b8-fabe-4822-881d-d776174dd6ec-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:29.472273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.472239 5607 generic.go:356] "Generic (PLEG): container finished" podID="375990b8-fabe-4822-881d-d776174dd6ec" containerID="c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5" exitCode=0 Oct 30 22:53:29.472455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.472297 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" Oct 30 22:53:29.472455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.472308 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" event={"ID":"375990b8-fabe-4822-881d-d776174dd6ec","Type":"ContainerDied","Data":"c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5"} Oct 30 22:53:29.472455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.472348 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548" event={"ID":"375990b8-fabe-4822-881d-d776174dd6ec","Type":"ContainerDied","Data":"0476ab24e11b3c6cf892b3156556eecbd5f51ecb3ffd78a4b3671079ced118a1"} Oct 30 22:53:29.472455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.472371 5607 scope.go:122] "RemoveContainer" containerID="c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5" Oct 30 22:53:29.482413 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.482398 5607 scope.go:122] "RemoveContainer" containerID="c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5" Oct 30 22:53:29.482662 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:29.482642 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5\": container with ID starting with c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5 not found: ID does not exist" containerID="c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5" Oct 30 22:53:29.482717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.482670 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5"} err="failed to get container status \"c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5\": rpc error: code = NotFound desc = could not find container \"c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5\": container with ID starting with c794c762faf903c9d0286e50783b1144ca03eddd482ffc51cb32d4edbc8c8fb5 not found: ID does not exist" Oct 30 22:53:29.641402 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.641375 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548"] Oct 30 22:53:29.692301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:29.692277 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e524ce-566b866699-kh548"] Oct 30 22:53:30.021724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:30.021696 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:30.022098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:30.021736 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:30.870302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:30.870274 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375990b8-fabe-4822-881d-d776174dd6ec" path="/var/lib/kubelet/pods/375990b8-fabe-4822-881d-d776174dd6ec/volumes" Oct 30 22:53:31.530048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.530015 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx"] Oct 30 22:53:31.530484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.530202 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" podUID="eec8660c-601d-4bdc-a880-4e6d9e2c1415" containerName="maas-api" containerID="cri-o://30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc" gracePeriod=30 Oct 30 22:53:31.802494 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.802429 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:53:31.841794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.841767 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\") pod \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " Oct 30 22:53:31.841928 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.841875 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w\" (UniqueName: \"kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w\") pod \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " Oct 30 22:53:31.841928 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.841921 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs\") pod \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " Oct 30 22:53:31.842083 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.842013 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca\") pod \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\" (UID: \"eec8660c-601d-4bdc-a880-4e6d9e2c1415\") " Oct 30 22:53:31.842415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.842392 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca" pod "eec8660c-601d-4bdc-a880-4e6d9e2c1415" (UID: "eec8660c-601d-4bdc-a880-4e6d9e2c1415"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:31.842533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.842511 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec8660c-601d-4bdc-a880-4e6d9e2c1415-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:31.844150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.844129 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w" pod "eec8660c-601d-4bdc-a880-4e6d9e2c1415" (UID: "eec8660c-601d-4bdc-a880-4e6d9e2c1415"). InnerVolumeSpecName "kube-api-access-sns7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:31.844649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.844618 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls" pod "eec8660c-601d-4bdc-a880-4e6d9e2c1415" (UID: "eec8660c-601d-4bdc-a880-4e6d9e2c1415"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:31.844649 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.844629 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs" pod "eec8660c-601d-4bdc-a880-4e6d9e2c1415" (UID: "eec8660c-601d-4bdc-a880-4e6d9e2c1415"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:31.943287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.943254 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:31.943287 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.943287 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sns7w\" (UniqueName: \"kubernetes.io/projected/eec8660c-601d-4bdc-a880-4e6d9e2c1415-kube-api-access-sns7w\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:31.943488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:31.943301 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eec8660c-601d-4bdc-a880-4e6d9e2c1415-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:32.492725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.492689 5607 generic.go:356] "Generic (PLEG): container finished" podID="eec8660c-601d-4bdc-a880-4e6d9e2c1415" containerID="30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc" exitCode=0 Oct 30 22:53:32.492914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.492738 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" Oct 30 22:53:32.492914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.492780 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" event={"ID":"eec8660c-601d-4bdc-a880-4e6d9e2c1415","Type":"ContainerDied","Data":"30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc"} Oct 30 22:53:32.492914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.492834 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx" event={"ID":"eec8660c-601d-4bdc-a880-4e6d9e2c1415","Type":"ContainerDied","Data":"3b3fa9c85d05ed1d667d36c61a729cdea4a24cadb6875f20743f28ed41912ea5"} Oct 30 22:53:32.492914 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.492862 5607 scope.go:122] "RemoveContainer" containerID="30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc" Oct 30 22:53:32.503527 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.503510 5607 scope.go:122] "RemoveContainer" containerID="30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc" Oct 30 22:53:32.503774 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:32.503755 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc\": container with ID starting with 30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc not found: ID does not exist" containerID="30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc" Oct 30 22:53:32.503847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.503784 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc"} err="failed to get container status \"30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc\": rpc error: code = NotFound desc = could not find container \"30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc\": container with ID starting with 30274aaf38364f73a0ea1614adc0c3e39b21d8262a790e560edfb06598b9bfcc not found: ID does not exist" Oct 30 22:53:32.663648 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.663619 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx"] Oct 30 22:53:32.715063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.715030 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-421286-5dc87674b9-zqkcx"] Oct 30 22:53:32.869880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:32.869806 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec8660c-601d-4bdc-a880-4e6d9e2c1415" path="/var/lib/kubelet/pods/eec8660c-601d-4bdc-a880-4e6d9e2c1415/volumes" Oct 30 22:53:40.021048 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:40.021004 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:40.021448 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:40.021018 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:53:40.410194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:40.410124 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2"] Oct 30 22:53:40.410355 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:40.410324 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" containerID="cri-o://1c184760f74456f6f8554e56a918e1d93d6073a118dd873fb74454e654dd90ad" gracePeriod=30 Oct 30 22:53:43.540772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:43.540738 5607 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:53:43.541288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:43.540803 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:53:45.573154 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.573116 5607 generic.go:356] "Generic (PLEG): container finished" podID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerID="1c184760f74456f6f8554e56a918e1d93d6073a118dd873fb74454e654dd90ad" exitCode=0 Oct 30 22:53:45.573567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.573189 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" event={"ID":"f2a5bbc0-5baf-43db-a70a-ea81a102c270","Type":"ContainerDied","Data":"1c184760f74456f6f8554e56a918e1d93d6073a118dd873fb74454e654dd90ad"} Oct 30 22:53:45.673825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.673772 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:53:45.783480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783457 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783506 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783606 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783642 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783678 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783733 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783781 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783852 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.783926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783892 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.783926 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.783901 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket\") pod \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\" (UID: \"f2a5bbc0-5baf-43db-a70a-ea81a102c270\") " Oct 30 22:53:45.784180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784034 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.784180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784148 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.784301 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784260 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.784365 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784338 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:45.784515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784453 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.784515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784478 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.784515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784494 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.784515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784507 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.784515 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.784519 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.786175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.786149 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:45.786284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.786262 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:53:45.786344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.786300 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:45.786662 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.786641 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx" pod "f2a5bbc0-5baf-43db-a70a-ea81a102c270" (UID: "f2a5bbc0-5baf-43db-a70a-ea81a102c270"). InnerVolumeSpecName "kube-api-access-j6lwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:45.885609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.885583 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.885609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.885607 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.885776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.885617 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a5bbc0-5baf-43db-a70a-ea81a102c270-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:45.885776 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:45.885626 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6lwx\" (UniqueName: \"kubernetes.io/projected/f2a5bbc0-5baf-43db-a70a-ea81a102c270-kube-api-access-j6lwx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:46.551884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.551859 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:53:46.580925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.580894 5607 generic.go:356] "Generic (PLEG): container finished" podID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerID="284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316" exitCode=0 Oct 30 22:53:46.581449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.580980 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" Oct 30 22:53:46.581449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.580986 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" event={"ID":"7899e2f7-a654-490b-aa02-af4124d6b43b","Type":"ContainerDied","Data":"284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316"} Oct 30 22:53:46.581449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.581046 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n" event={"ID":"7899e2f7-a654-490b-aa02-af4124d6b43b","Type":"ContainerDied","Data":"d410270449d883474cd047b1cb59133efd4e9402b7bef89aec9beabf976687f5"} Oct 30 22:53:46.581449 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.581075 5607 scope.go:122] "RemoveContainer" containerID="284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316" Oct 30 22:53:46.583111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.583088 5607 generic.go:356] "Generic (PLEG): container finished" podID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerID="53de3ccb94a6a5e574606e537c8c29d423d37450376f958ee855afa1f8adfda0" exitCode=0 Oct 30 22:53:46.583250 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.583160 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" event={"ID":"6c3dd339-b978-4b7a-998b-587e00c82bca","Type":"ContainerDied","Data":"53de3ccb94a6a5e574606e537c8c29d423d37450376f958ee855afa1f8adfda0"} Oct 30 22:53:46.584716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.584694 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" Oct 30 22:53:46.584829 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.584710 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2" event={"ID":"f2a5bbc0-5baf-43db-a70a-ea81a102c270","Type":"ContainerDied","Data":"c2a1195f2b7d27ccb3720cd65f8af4b5d3453014f6996f0f6a58d6bfa10e8aff"} Oct 30 22:53:46.592591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.592568 5607 scope.go:122] "RemoveContainer" containerID="284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316" Oct 30 22:53:46.592913 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:46.592875 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316\": container with ID starting with 284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316 not found: ID does not exist" containerID="284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316" Oct 30 22:53:46.593018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.592908 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316"} err="failed to get container status \"284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316\": rpc error: code = NotFound desc = could not find container \"284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316\": container with ID starting with 284a0ae6b0b694fe5ae4c3c1692c7da6937c615c5290ee28efeca52dd203c316 not found: ID does not exist" Oct 30 22:53:46.593018 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.592929 5607 scope.go:122] "RemoveContainer" containerID="1c184760f74456f6f8554e56a918e1d93d6073a118dd873fb74454e654dd90ad" Oct 30 22:53:46.668872 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.668851 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:53:46.693814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.693783 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr\" (UniqueName: \"kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr\") pod \"7899e2f7-a654-490b-aa02-af4124d6b43b\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " Oct 30 22:53:46.694198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.693889 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume\") pod \"7899e2f7-a654-490b-aa02-af4124d6b43b\" (UID: \"7899e2f7-a654-490b-aa02-af4124d6b43b\") " Oct 30 22:53:46.694292 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.694268 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume" pod "7899e2f7-a654-490b-aa02-af4124d6b43b" (UID: "7899e2f7-a654-490b-aa02-af4124d6b43b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:46.694482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.694458 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7899e2f7-a654-490b-aa02-af4124d6b43b-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:46.696053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.696028 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr" pod "7899e2f7-a654-490b-aa02-af4124d6b43b" (UID: "7899e2f7-a654-490b-aa02-af4124d6b43b"). InnerVolumeSpecName "kube-api-access-slrdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:46.795876 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.795801 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt\" (UniqueName: \"kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt\") pod \"6c3dd339-b978-4b7a-998b-587e00c82bca\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " Oct 30 22:53:46.796046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.795945 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume\") pod \"6c3dd339-b978-4b7a-998b-587e00c82bca\" (UID: \"6c3dd339-b978-4b7a-998b-587e00c82bca\") " Oct 30 22:53:46.796376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.796349 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrdr\" (UniqueName: \"kubernetes.io/projected/7899e2f7-a654-490b-aa02-af4124d6b43b-kube-api-access-slrdr\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:46.796376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.796349 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume" pod "6c3dd339-b978-4b7a-998b-587e00c82bca" (UID: "6c3dd339-b978-4b7a-998b-587e00c82bca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:46.798053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.798033 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt" pod "6c3dd339-b978-4b7a-998b-587e00c82bca" (UID: "6c3dd339-b978-4b7a-998b-587e00c82bca"). InnerVolumeSpecName "kube-api-access-l8gzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:46.897637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.897612 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3dd339-b978-4b7a-998b-587e00c82bca-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:46.897637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.897638 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gzt\" (UniqueName: \"kubernetes.io/projected/6c3dd339-b978-4b7a-998b-587e00c82bca-kube-api-access-l8gzt\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:46.956531 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:46.956507 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2"] Oct 30 22:53:47.007360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.007335 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-lcxv2"] Oct 30 22:53:47.072661 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.072597 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n"] Oct 30 22:53:47.076160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.076140 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-435923-5f7478d95c-9dc6n"] Oct 30 22:53:47.594345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.594317 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" Oct 30 22:53:47.594845 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.594325 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77" event={"ID":"6c3dd339-b978-4b7a-998b-587e00c82bca","Type":"ContainerDied","Data":"dda4d53e55e43734ae84dd7613763d5003d42c3ad19c7b738f2e24f3abe1f343"} Oct 30 22:53:47.595017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.594871 5607 scope.go:122] "RemoveContainer" containerID="53de3ccb94a6a5e574606e537c8c29d423d37450376f958ee855afa1f8adfda0" Oct 30 22:53:47.763137 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.763104 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77"] Oct 30 22:53:47.814002 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:47.813974 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-435923-5cbcfb6f79-hvr77"] Oct 30 22:53:48.869224 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:48.869192 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" path="/var/lib/kubelet/pods/6c3dd339-b978-4b7a-998b-587e00c82bca/volumes" Oct 30 22:53:48.869609 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:48.869566 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" path="/var/lib/kubelet/pods/7899e2f7-a654-490b-aa02-af4124d6b43b/volumes" Oct 30 22:53:48.869975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:53:48.869945 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" path="/var/lib/kubelet/pods/f2a5bbc0-5baf-43db-a70a-ea81a102c270/volumes" Oct 30 22:53:52.863385 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:53:52.863346 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:54:22.738409 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:22.738378 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq"] Oct 30 22:54:22.846291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:22.846262 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh"] Oct 30 22:54:22.865523 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:54:22.865490 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:54:27.739321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:27.739263 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerName="payload-processing" containerID="cri-o://f9789cc442295ab0a6691509ad05dc1dc80503f2275239b42cfc561aa77eb123" gracePeriod=25 Oct 30 22:54:27.846741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:27.846703 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" podUID="2f0b9648-ca39-4146-a501-040489a211c2" containerName="payload-pre-processing" containerID="cri-o://0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2" gracePeriod=25 Oct 30 22:54:31.568161 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:31.568123 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" podUID="2f0b9648-ca39-4146-a501-040489a211c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:54:31.568547 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:31.568130 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:54:41.567731 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:41.567641 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" podUID="2f0b9648-ca39-4146-a501-040489a211c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:54:41.568169 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:41.567641 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:54:46.240408 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:46.240377 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6"] Oct 30 22:54:46.240840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:46.240575 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" containerID="cri-o://fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34" gracePeriod=30 Oct 30 22:54:47.955260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:47.955087 5607 generic.go:356] "Generic (PLEG): container finished" podID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerID="f9789cc442295ab0a6691509ad05dc1dc80503f2275239b42cfc561aa77eb123" exitCode=0 Oct 30 22:54:47.955260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:47.955182 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" event={"ID":"549f9160-eeb0-40e9-81f8-dfeecc639d62","Type":"ContainerDied","Data":"f9789cc442295ab0a6691509ad05dc1dc80503f2275239b42cfc561aa77eb123"} Oct 30 22:54:48.013766 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.013702 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 22:54:48.099062 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.099035 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume\") pod \"549f9160-eeb0-40e9-81f8-dfeecc639d62\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " Oct 30 22:54:48.099194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.099110 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46\" (UniqueName: \"kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46\") pod \"549f9160-eeb0-40e9-81f8-dfeecc639d62\" (UID: \"549f9160-eeb0-40e9-81f8-dfeecc639d62\") " Oct 30 22:54:48.099455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.099426 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume" pod "549f9160-eeb0-40e9-81f8-dfeecc639d62" (UID: "549f9160-eeb0-40e9-81f8-dfeecc639d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:48.099560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.099537 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/549f9160-eeb0-40e9-81f8-dfeecc639d62-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:48.101273 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.101251 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46" pod "549f9160-eeb0-40e9-81f8-dfeecc639d62" (UID: "549f9160-eeb0-40e9-81f8-dfeecc639d62"). InnerVolumeSpecName "kube-api-access-f9h46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:48.102539 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.102519 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 22:54:48.200180 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.200156 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw\" (UniqueName: \"kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw\") pod \"2f0b9648-ca39-4146-a501-040489a211c2\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " Oct 30 22:54:48.200313 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.200208 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume\") pod \"2f0b9648-ca39-4146-a501-040489a211c2\" (UID: \"2f0b9648-ca39-4146-a501-040489a211c2\") " Oct 30 22:54:48.200576 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.200558 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h46\" (UniqueName: \"kubernetes.io/projected/549f9160-eeb0-40e9-81f8-dfeecc639d62-kube-api-access-f9h46\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:48.200681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.200583 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume" pod "2f0b9648-ca39-4146-a501-040489a211c2" (UID: "2f0b9648-ca39-4146-a501-040489a211c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:48.202343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.202325 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw" pod "2f0b9648-ca39-4146-a501-040489a211c2" (UID: "2f0b9648-ca39-4146-a501-040489a211c2"). InnerVolumeSpecName "kube-api-access-fmkfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:48.301183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.301115 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkfw\" (UniqueName: \"kubernetes.io/projected/2f0b9648-ca39-4146-a501-040489a211c2-kube-api-access-fmkfw\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:48.301183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.301148 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0b9648-ca39-4146-a501-040489a211c2-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:48.962209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.962177 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" Oct 30 22:54:48.962602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.962178 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq" event={"ID":"549f9160-eeb0-40e9-81f8-dfeecc639d62","Type":"ContainerDied","Data":"f892ccb7794676795bd830b42c8949c4b47d8d9e2447338916990858a9e30c7b"} Oct 30 22:54:48.962602 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.962318 5607 scope.go:122] "RemoveContainer" containerID="f9789cc442295ab0a6691509ad05dc1dc80503f2275239b42cfc561aa77eb123" Oct 30 22:54:48.963724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.963520 5607 generic.go:356] "Generic (PLEG): container finished" podID="2f0b9648-ca39-4146-a501-040489a211c2" containerID="0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2" exitCode=0 Oct 30 22:54:48.963724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.963572 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" event={"ID":"2f0b9648-ca39-4146-a501-040489a211c2","Type":"ContainerDied","Data":"0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2"} Oct 30 22:54:48.963724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.963611 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" event={"ID":"2f0b9648-ca39-4146-a501-040489a211c2","Type":"ContainerDied","Data":"26e67150ed7077d0b3b380efa117d022c4893961d7a6aa04de84e3f5cea7a96c"} Oct 30 22:54:48.963724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.963703 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh" Oct 30 22:54:48.971687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.971668 5607 scope.go:122] "RemoveContainer" containerID="0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2" Oct 30 22:54:48.980035 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.980014 5607 scope.go:122] "RemoveContainer" containerID="0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2" Oct 30 22:54:48.980294 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:54:48.980274 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2\": container with ID starting with 0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2 not found: ID does not exist" containerID="0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2" Oct 30 22:54:48.980354 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:48.980306 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2"} err="failed to get container status \"0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2\": rpc error: code = NotFound desc = could not find container \"0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2\": container with ID starting with 0a8e3848df418bb8b316895ea59b2b3390773b6f71c73ce2c9214bb7aa7f06f2 not found: ID does not exist" Oct 30 22:54:49.130522 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:49.130462 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh"] Oct 30 22:54:49.134335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:49.134317 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b47e8d-6d9ff66fcf-kzjsh"] Oct 30 22:54:49.246702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:49.246679 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq"] Oct 30 22:54:49.251228 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:49.251209 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b47e8d-555f77c5b5-9qnmq"] Oct 30 22:54:50.869516 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:50.869483 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0b9648-ca39-4146-a501-040489a211c2" path="/var/lib/kubelet/pods/2f0b9648-ca39-4146-a501-040489a211c2/volumes" Oct 30 22:54:50.869896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:50.869837 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" path="/var/lib/kubelet/pods/549f9160-eeb0-40e9-81f8-dfeecc639d62/volumes" Oct 30 22:54:51.488264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.488245 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:54:51.529235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529167 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529249 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529376 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529304 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529409 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529442 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529504 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529476 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529512 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529564 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:51.529667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529592 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529667 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529644 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data\") pod \"f1afb645-4dd0-4d16-8899-e65634921b01\" (UID: \"f1afb645-4dd0-4d16-8899-e65634921b01\") " Oct 30 22:54:51.529881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529759 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:51.529881 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.529845 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:51.530033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.530004 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:51.530276 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.530254 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:51.530411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.530388 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.530497 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.530419 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1afb645-4dd0-4d16-8899-e65634921b01-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.530497 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.530435 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.530497 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.530450 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.531712 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.531686 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:51.531821 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.531791 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:51.531986 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.531934 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:54:51.532843 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.532826 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx" pod "f1afb645-4dd0-4d16-8899-e65634921b01" (UID: "f1afb645-4dd0-4d16-8899-e65634921b01"). InnerVolumeSpecName "kube-api-access-68rkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:51.631589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.631562 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.631589 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.631586 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rkx\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-kube-api-access-68rkx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.631778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.631596 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1afb645-4dd0-4d16-8899-e65634921b01-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.631778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.631605 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1afb645-4dd0-4d16-8899-e65634921b01-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.631778 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.631614 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1afb645-4dd0-4d16-8899-e65634921b01-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:51.982772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.982738 5607 generic.go:356] "Generic (PLEG): container finished" podID="f1afb645-4dd0-4d16-8899-e65634921b01" containerID="fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34" exitCode=0 Oct 30 22:54:51.983257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.982790 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" Oct 30 22:54:51.983257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.982792 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" event={"ID":"f1afb645-4dd0-4d16-8899-e65634921b01","Type":"ContainerDied","Data":"fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34"} Oct 30 22:54:51.983257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.982848 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6" event={"ID":"f1afb645-4dd0-4d16-8899-e65634921b01","Type":"ContainerDied","Data":"9b9113e2dae7fba49e5b5e6700dbaa14f593cadba07e2ca02a28aff0418d5213"} Oct 30 22:54:51.983257 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.982877 5607 scope.go:122] "RemoveContainer" containerID="fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34" Oct 30 22:54:51.993063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.993045 5607 scope.go:122] "RemoveContainer" containerID="fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34" Oct 30 22:54:51.993353 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:54:51.993333 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34\": container with ID starting with fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34 not found: ID does not exist" containerID="fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34" Oct 30 22:54:51.993411 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:51.993360 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34"} err="failed to get container status \"fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34\": rpc error: code = NotFound desc = could not find container \"fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34\": container with ID starting with fd3a6d3e32a8e99619a43841c86a41e9af8275effb1ef10924c2402489badd34 not found: ID does not exist" Oct 30 22:54:52.154725 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:52.154692 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6"] Oct 30 22:54:52.165336 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:52.165308 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-sdtd6"] Oct 30 22:54:52.862725 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:54:52.862688 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:54:52.870460 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:52.870431 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" path="/var/lib/kubelet/pods/f1afb645-4dd0-4d16-8899-e65634921b01/volumes" Oct 30 22:54:52.979919 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:52.979885 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp"] Oct 30 22:54:52.980165 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:52.980142 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" podUID="09f8a61b-d445-4b2d-b66c-e37b18c93114" containerName="maas-api" containerID="cri-o://cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1" gracePeriod=30 Oct 30 22:54:53.250369 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.250345 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:54:53.347308 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.347277 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq\" (UniqueName: \"kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq\") pod \"09f8a61b-d445-4b2d-b66c-e37b18c93114\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " Oct 30 22:54:53.347501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.347324 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca\") pod \"09f8a61b-d445-4b2d-b66c-e37b18c93114\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " Oct 30 22:54:53.347501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.347435 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs\") pod \"09f8a61b-d445-4b2d-b66c-e37b18c93114\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " Oct 30 22:54:53.347501 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.347498 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\") pod \"09f8a61b-d445-4b2d-b66c-e37b18c93114\" (UID: \"09f8a61b-d445-4b2d-b66c-e37b18c93114\") " Oct 30 22:54:53.347732 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.347712 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca" pod "09f8a61b-d445-4b2d-b66c-e37b18c93114" (UID: "09f8a61b-d445-4b2d-b66c-e37b18c93114"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:53.347939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.347920 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09f8a61b-d445-4b2d-b66c-e37b18c93114-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.350112 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.350089 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq" pod "09f8a61b-d445-4b2d-b66c-e37b18c93114" (UID: "09f8a61b-d445-4b2d-b66c-e37b18c93114"). InnerVolumeSpecName "kube-api-access-t2kxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:53.350206 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.350120 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs" pod "09f8a61b-d445-4b2d-b66c-e37b18c93114" (UID: "09f8a61b-d445-4b2d-b66c-e37b18c93114"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:53.350548 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.350529 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls" pod "09f8a61b-d445-4b2d-b66c-e37b18c93114" (UID: "09f8a61b-d445-4b2d-b66c-e37b18c93114"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:53.406455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.406424 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p"] Oct 30 22:54:53.449043 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.449017 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2kxq\" (UniqueName: \"kubernetes.io/projected/09f8a61b-d445-4b2d-b66c-e37b18c93114-kube-api-access-t2kxq\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.449188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.449046 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:53.449188 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:53.449061 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09f8a61b-d445-4b2d-b66c-e37b18c93114-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:54.014370 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.014332 5607 generic.go:356] "Generic (PLEG): container finished" podID="09f8a61b-d445-4b2d-b66c-e37b18c93114" containerID="cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1" exitCode=0 Oct 30 22:54:54.014642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.014386 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" Oct 30 22:54:54.014642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.014415 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" event={"ID":"09f8a61b-d445-4b2d-b66c-e37b18c93114","Type":"ContainerDied","Data":"cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1"} Oct 30 22:54:54.014642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.014471 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp" event={"ID":"09f8a61b-d445-4b2d-b66c-e37b18c93114","Type":"ContainerDied","Data":"96cf1dba484ab1ed99467da03f20cf3a389bb003b8295e1a41a7b79c25400926"} Oct 30 22:54:54.014642 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.014495 5607 scope.go:122] "RemoveContainer" containerID="cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1" Oct 30 22:54:54.024824 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.024808 5607 scope.go:122] "RemoveContainer" containerID="cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1" Oct 30 22:54:54.025118 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:54:54.025097 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1\": container with ID starting with cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1 not found: ID does not exist" containerID="cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1" Oct 30 22:54:54.025209 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.025131 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1"} err="failed to get container status \"cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1\": rpc error: code = NotFound desc = could not find container \"cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1\": container with ID starting with cd3b47f24ac27ebfdef9a066d37416e56cb29cca47f293b8edc20e83844440c1 not found: ID does not exist" Oct 30 22:54:54.183295 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.183270 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp"] Oct 30 22:54:54.191568 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.191547 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9717a8-8b74c4797-j5vtp"] Oct 30 22:54:54.870020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:54.869947 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f8a61b-d445-4b2d-b66c-e37b18c93114" path="/var/lib/kubelet/pods/09f8a61b-d445-4b2d-b66c-e37b18c93114/volumes" Oct 30 22:54:58.407306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:54:58.407263 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerName="payload-processing" containerID="cri-o://11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192" gracePeriod=25 Oct 30 22:55:07.381623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:07.381578 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:55:17.382157 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:17.382110 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:55:18.678893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.678828 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:55:18.698017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.697993 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9\" (UniqueName: \"kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9\") pod \"45326de7-55f6-4593-9ae2-827a3f87eefb\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " Oct 30 22:55:18.698136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.698086 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume\") pod \"45326de7-55f6-4593-9ae2-827a3f87eefb\" (UID: \"45326de7-55f6-4593-9ae2-827a3f87eefb\") " Oct 30 22:55:18.698484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.698460 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume" pod "45326de7-55f6-4593-9ae2-827a3f87eefb" (UID: "45326de7-55f6-4593-9ae2-827a3f87eefb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:18.700402 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.700377 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9" pod "45326de7-55f6-4593-9ae2-827a3f87eefb" (UID: "45326de7-55f6-4593-9ae2-827a3f87eefb"). InnerVolumeSpecName "kube-api-access-s72g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:18.799946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.799915 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45326de7-55f6-4593-9ae2-827a3f87eefb-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:18.799946 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:18.799968 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72g9\" (UniqueName: \"kubernetes.io/projected/45326de7-55f6-4593-9ae2-827a3f87eefb-kube-api-access-s72g9\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:19.147232 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.147197 5607 generic.go:356] "Generic (PLEG): container finished" podID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerID="11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192" exitCode=0 Oct 30 22:55:19.147379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.147241 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" Oct 30 22:55:19.147379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.147283 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" event={"ID":"45326de7-55f6-4593-9ae2-827a3f87eefb","Type":"ContainerDied","Data":"11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192"} Oct 30 22:55:19.147379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.147335 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p" event={"ID":"45326de7-55f6-4593-9ae2-827a3f87eefb","Type":"ContainerDied","Data":"6f52827129ee45a07c3a048a9f1a5de2b5e900c05ec04b34b90618f928d00eca"} Oct 30 22:55:19.147379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.147357 5607 scope.go:122] "RemoveContainer" containerID="11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192" Oct 30 22:55:19.156490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.156472 5607 scope.go:122] "RemoveContainer" containerID="11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192" Oct 30 22:55:19.156728 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:55:19.156712 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192\": container with ID starting with 11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192 not found: ID does not exist" containerID="11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192" Oct 30 22:55:19.156823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.156738 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192"} err="failed to get container status \"11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192\": rpc error: code = NotFound desc = could not find container \"11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192\": container with ID starting with 11a7c3287b7bc03093c8a8ad774b2a79014dd4e1086f55a71b3aa671f9044192 not found: ID does not exist" Oct 30 22:55:19.315998 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.315968 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p"] Oct 30 22:55:19.320024 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:19.320004 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9717a8-6c58658699-zfk7p"] Oct 30 22:55:20.869849 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:20.869813 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" path="/var/lib/kubelet/pods/45326de7-55f6-4593-9ae2-827a3f87eefb/volumes" Oct 30 22:55:22.862179 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:55:22.862138 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:55:32.572738 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.572704 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm"] Oct 30 22:55:32.573329 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.572893 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" podUID="6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" containerName="maas-api" containerID="cri-o://00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022" gracePeriod=30 Oct 30 22:55:32.827918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.827854 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:55:32.932156 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.932128 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\") pod \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " Oct 30 22:55:32.932344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.932228 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp\" (UniqueName: \"kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp\") pod \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " Oct 30 22:55:32.932344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.932250 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs\") pod \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " Oct 30 22:55:32.932344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.932287 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca\") pod \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\" (UID: \"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d\") " Oct 30 22:55:32.933051 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.932854 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca" pod "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" (UID: "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:32.935398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.935369 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls" pod "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" (UID: "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:32.935398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.935393 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp" pod "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" (UID: "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d"). InnerVolumeSpecName "kube-api-access-hs7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:32.935592 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:32.935536 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs" pod "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" (UID: "6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:33.034320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.034289 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:33.034320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.034317 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7xp\" (UniqueName: \"kubernetes.io/projected/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-kube-api-access-hs7xp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:33.034320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.034328 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:33.034554 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.034337 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:33.220599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.220565 5607 generic.go:356] "Generic (PLEG): container finished" podID="6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" containerID="00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022" exitCode=0 Oct 30 22:55:33.220768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.220623 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" Oct 30 22:55:33.220768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.220658 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" event={"ID":"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d","Type":"ContainerDied","Data":"00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022"} Oct 30 22:55:33.220768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.220716 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm" event={"ID":"6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d","Type":"ContainerDied","Data":"f6e2c3d06889b9dfbed5e9048dfcd3e6dff125c9149903a6b3ce0a4e8d96cac5"} Oct 30 22:55:33.220768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.220741 5607 scope.go:122] "RemoveContainer" containerID="00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022" Oct 30 22:55:33.230911 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.230895 5607 scope.go:122] "RemoveContainer" containerID="00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022" Oct 30 22:55:33.231211 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:55:33.231191 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022\": container with ID starting with 00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022 not found: ID does not exist" containerID="00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022" Oct 30 22:55:33.231268 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.231219 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022"} err="failed to get container status \"00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022\": rpc error: code = NotFound desc = could not find container \"00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022\": container with ID starting with 00ce2fe426db37656ab590331bc7498f9882f3bdb193db7a9c290dbca1af0022 not found: ID does not exist" Oct 30 22:55:33.390150 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.390126 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm"] Oct 30 22:55:33.396788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.396770 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9626a6-7d448c6b9d-cx7rm"] Oct 30 22:55:33.645272 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.645183 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g"] Oct 30 22:55:33.645769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.645492 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" podUID="2216473b-6bc3-4d55-b118-9bc2de085859" containerName="maas-api" containerID="cri-o://23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5" gracePeriod=30 Oct 30 22:55:33.913398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:33.913375 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:55:34.047224 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.047173 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88\" (UniqueName: \"kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88\") pod \"2216473b-6bc3-4d55-b118-9bc2de085859\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " Oct 30 22:55:34.047427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.047349 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\") pod \"2216473b-6bc3-4d55-b118-9bc2de085859\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " Oct 30 22:55:34.047427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.047415 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs\") pod \"2216473b-6bc3-4d55-b118-9bc2de085859\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " Oct 30 22:55:34.047580 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.047443 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca\") pod \"2216473b-6bc3-4d55-b118-9bc2de085859\" (UID: \"2216473b-6bc3-4d55-b118-9bc2de085859\") " Oct 30 22:55:34.048320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.048259 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca" pod "2216473b-6bc3-4d55-b118-9bc2de085859" (UID: "2216473b-6bc3-4d55-b118-9bc2de085859"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:34.050233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.050207 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls" pod "2216473b-6bc3-4d55-b118-9bc2de085859" (UID: "2216473b-6bc3-4d55-b118-9bc2de085859"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:34.051214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.051184 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88" pod "2216473b-6bc3-4d55-b118-9bc2de085859" (UID: "2216473b-6bc3-4d55-b118-9bc2de085859"). InnerVolumeSpecName "kube-api-access-gdm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:34.055344 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.055318 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs" pod "2216473b-6bc3-4d55-b118-9bc2de085859" (UID: "2216473b-6bc3-4d55-b118-9bc2de085859"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:34.149429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.149335 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdm88\" (UniqueName: \"kubernetes.io/projected/2216473b-6bc3-4d55-b118-9bc2de085859-kube-api-access-gdm88\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:34.149429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.149378 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:34.149429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.149394 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2216473b-6bc3-4d55-b118-9bc2de085859-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:34.149429 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.149406 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2216473b-6bc3-4d55-b118-9bc2de085859-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:34.227685 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.227652 5607 generic.go:356] "Generic (PLEG): container finished" podID="2216473b-6bc3-4d55-b118-9bc2de085859" containerID="23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5" exitCode=0 Oct 30 22:55:34.227867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.227707 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" Oct 30 22:55:34.227867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.227743 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" event={"ID":"2216473b-6bc3-4d55-b118-9bc2de085859","Type":"ContainerDied","Data":"23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5"} Oct 30 22:55:34.227867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.227803 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g" event={"ID":"2216473b-6bc3-4d55-b118-9bc2de085859","Type":"ContainerDied","Data":"5e089462b36bcce57e111c00e14c3239856448cccfffca77954f59a9d11d3a3b"} Oct 30 22:55:34.227867 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.227831 5607 scope.go:122] "RemoveContainer" containerID="23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5" Oct 30 22:55:34.237814 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.237798 5607 scope.go:122] "RemoveContainer" containerID="23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5" Oct 30 22:55:34.238120 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:55:34.238099 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5\": container with ID starting with 23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5 not found: ID does not exist" containerID="23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5" Oct 30 22:55:34.238203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.238142 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5"} err="failed to get container status \"23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5\": rpc error: code = NotFound desc = could not find container \"23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5\": container with ID starting with 23d17f863ffaae1af00a2a4a624d91784bf9090d4c2cf6de97aebe58e018e7f5 not found: ID does not exist" Oct 30 22:55:34.400073 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.399985 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g"] Oct 30 22:55:34.401723 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.401702 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-68f80c-6556f97544-b5c2g"] Oct 30 22:55:34.870537 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.870501 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2216473b-6bc3-4d55-b118-9bc2de085859" path="/var/lib/kubelet/pods/2216473b-6bc3-4d55-b118-9bc2de085859/volumes" Oct 30 22:55:34.871030 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:34.870984 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" path="/var/lib/kubelet/pods/6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d/volumes" Oct 30 22:55:52.862423 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:55:52.862380 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:55:56.743629 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:56.743591 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg"] Oct 30 22:55:56.858590 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:55:56.858559 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h"] Oct 30 22:56:01.744217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:01.744175 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerName="payload-processing" containerID="cri-o://fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3" gracePeriod=25 Oct 30 22:56:01.859505 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:01.859468 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerName="payload-pre-processing" containerID="cri-o://e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2" gracePeriod=25 Oct 30 22:56:05.532001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:05.531963 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:06.538567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:06.538532 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:15.532130 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:15.532048 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:16.538564 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:16.538526 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:22.036175 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.035824 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 22:56:22.134866 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.134837 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 22:56:22.191593 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.191568 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp\" (UniqueName: \"kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp\") pod \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " Oct 30 22:56:22.191751 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.191620 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume\") pod \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\" (UID: \"bfa9bead-f06b-4467-8d59-fa25bf5354ca\") " Oct 30 22:56:22.192017 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.191984 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume" pod "bfa9bead-f06b-4467-8d59-fa25bf5354ca" (UID: "bfa9bead-f06b-4467-8d59-fa25bf5354ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:22.192136 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.192106 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa9bead-f06b-4467-8d59-fa25bf5354ca-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:22.193994 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.193967 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp" pod "bfa9bead-f06b-4467-8d59-fa25bf5354ca" (UID: "bfa9bead-f06b-4467-8d59-fa25bf5354ca"). InnerVolumeSpecName "kube-api-access-5hhsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:22.292839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.292760 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq\" (UniqueName: \"kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq\") pod \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " Oct 30 22:56:22.292839 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.292829 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume\") pod \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\" (UID: \"afd9a3b5-8766-45f2-a471-68f8443cc2f6\") " Oct 30 22:56:22.293270 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.293246 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hhsp\" (UniqueName: \"kubernetes.io/projected/bfa9bead-f06b-4467-8d59-fa25bf5354ca-kube-api-access-5hhsp\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:22.293397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.293377 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume" pod "afd9a3b5-8766-45f2-a471-68f8443cc2f6" (UID: "afd9a3b5-8766-45f2-a471-68f8443cc2f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:22.295321 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.295301 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq" pod "afd9a3b5-8766-45f2-a471-68f8443cc2f6" (UID: "afd9a3b5-8766-45f2-a471-68f8443cc2f6"). InnerVolumeSpecName "kube-api-access-s7wdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:22.394675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.394650 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd9a3b5-8766-45f2-a471-68f8443cc2f6-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:22.394675 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.394675 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wdq\" (UniqueName: \"kubernetes.io/projected/afd9a3b5-8766-45f2-a471-68f8443cc2f6-kube-api-access-s7wdq\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:22.482412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.482377 5607 generic.go:356] "Generic (PLEG): container finished" podID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerID="fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3" exitCode=0 Oct 30 22:56:22.482560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.482428 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" Oct 30 22:56:22.482560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.482465 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" event={"ID":"bfa9bead-f06b-4467-8d59-fa25bf5354ca","Type":"ContainerDied","Data":"fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3"} Oct 30 22:56:22.482560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.482523 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg" event={"ID":"bfa9bead-f06b-4467-8d59-fa25bf5354ca","Type":"ContainerDied","Data":"94e4b62304328713a6d26d8c35be270c933cfedffc1936d327f68e7f467d3b8c"} Oct 30 22:56:22.482560 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.482550 5607 scope.go:122] "RemoveContainer" containerID="fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3" Oct 30 22:56:22.484094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.484069 5607 generic.go:356] "Generic (PLEG): container finished" podID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerID="e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2" exitCode=0 Oct 30 22:56:22.484233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.484113 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" event={"ID":"afd9a3b5-8766-45f2-a471-68f8443cc2f6","Type":"ContainerDied","Data":"e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2"} Oct 30 22:56:22.484233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.484127 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" Oct 30 22:56:22.484335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.484320 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h" event={"ID":"afd9a3b5-8766-45f2-a471-68f8443cc2f6","Type":"ContainerDied","Data":"73e786502c560372467d52c0584c0ed2a3863b82070655e8418a13eccc26e625"} Oct 30 22:56:22.491983 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.491927 5607 scope.go:122] "RemoveContainer" containerID="fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3" Oct 30 22:56:22.492278 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:22.492255 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3\": container with ID starting with fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3 not found: ID does not exist" containerID="fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3" Oct 30 22:56:22.492356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.492291 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3"} err="failed to get container status \"fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3\": rpc error: code = NotFound desc = could not find container \"fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3\": container with ID starting with fdfca3f52b44d55fe19d2fabeb29e2a28f56f8cd481783cd3960e990aa331ac3 not found: ID does not exist" Oct 30 22:56:22.492356 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.492315 5607 scope.go:122] "RemoveContainer" containerID="e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2" Oct 30 22:56:22.501381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.501365 5607 scope.go:122] "RemoveContainer" containerID="e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2" Oct 30 22:56:22.501665 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:22.501637 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2\": container with ID starting with e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2 not found: ID does not exist" containerID="e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2" Oct 30 22:56:22.501783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.501669 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2"} err="failed to get container status \"e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2\": rpc error: code = NotFound desc = could not find container \"e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2\": container with ID starting with e1904e0e07281815ea990ac17a0eb5208783f0a5c02bfcf50a2be7fec86db5f2 not found: ID does not exist" Oct 30 22:56:22.653571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.653518 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h"] Oct 30 22:56:22.660004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.659976 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7f66ab-747b4fcdb7-jb99h"] Oct 30 22:56:22.772293 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.772264 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg"] Oct 30 22:56:22.779415 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.779394 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7f66ab-6b68bd85cf-w9lbg"] Oct 30 22:56:22.862276 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:22.862246 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:56:22.869108 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.869086 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" path="/var/lib/kubelet/pods/afd9a3b5-8766-45f2-a471-68f8443cc2f6/volumes" Oct 30 22:56:22.869512 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:22.869498 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" path="/var/lib/kubelet/pods/bfa9bead-f06b-4467-8d59-fa25bf5354ca/volumes" Oct 30 22:56:27.331238 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:27.331190 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st"] Oct 30 22:56:27.441760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:27.441473 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx"] Oct 30 22:56:29.525049 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.525020 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9"] Oct 30 22:56:29.525518 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.525197 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" podUID="c397a2bb-d114-4394-ab83-5b3c0ba77994" containerName="maas-api" containerID="cri-o://72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f" gracePeriod=30 Oct 30 22:56:29.773658 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.773632 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:56:29.866746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.866682 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca\") pod \"c397a2bb-d114-4394-ab83-5b3c0ba77994\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " Oct 30 22:56:29.866746 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.866729 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs\") pod \"c397a2bb-d114-4394-ab83-5b3c0ba77994\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " Oct 30 22:56:29.866921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.866777 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv\" (UniqueName: \"kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv\") pod \"c397a2bb-d114-4394-ab83-5b3c0ba77994\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " Oct 30 22:56:29.866921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.866879 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\") pod \"c397a2bb-d114-4394-ab83-5b3c0ba77994\" (UID: \"c397a2bb-d114-4394-ab83-5b3c0ba77994\") " Oct 30 22:56:29.867077 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.867053 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca" pod "c397a2bb-d114-4394-ab83-5b3c0ba77994" (UID: "c397a2bb-d114-4394-ab83-5b3c0ba77994"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:29.867300 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.867278 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c397a2bb-d114-4394-ab83-5b3c0ba77994-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:29.869115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.869091 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs" pod "c397a2bb-d114-4394-ab83-5b3c0ba77994" (UID: "c397a2bb-d114-4394-ab83-5b3c0ba77994"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:29.869115 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.869105 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv" pod "c397a2bb-d114-4394-ab83-5b3c0ba77994" (UID: "c397a2bb-d114-4394-ab83-5b3c0ba77994"). InnerVolumeSpecName "kube-api-access-zc8wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:29.869261 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.869110 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls" pod "c397a2bb-d114-4394-ab83-5b3c0ba77994" (UID: "c397a2bb-d114-4394-ab83-5b3c0ba77994"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:29.968770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.968740 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:29.968770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.968766 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c397a2bb-d114-4394-ab83-5b3c0ba77994-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:29.968988 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:29.968776 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8wv\" (UniqueName: \"kubernetes.io/projected/c397a2bb-d114-4394-ab83-5b3c0ba77994-kube-api-access-zc8wv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:30.529884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.529843 5607 generic.go:356] "Generic (PLEG): container finished" podID="c397a2bb-d114-4394-ab83-5b3c0ba77994" containerID="72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f" exitCode=0 Oct 30 22:56:30.529884 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.529887 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" Oct 30 22:56:30.530352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.529927 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" event={"ID":"c397a2bb-d114-4394-ab83-5b3c0ba77994","Type":"ContainerDied","Data":"72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f"} Oct 30 22:56:30.530352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.529995 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9" event={"ID":"c397a2bb-d114-4394-ab83-5b3c0ba77994","Type":"ContainerDied","Data":"552f5c2c002d22b2600af716b567ed432e94956d922657251303634afb83302c"} Oct 30 22:56:30.530352 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.530018 5607 scope.go:122] "RemoveContainer" containerID="72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f" Oct 30 22:56:30.540166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.540146 5607 scope.go:122] "RemoveContainer" containerID="72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f" Oct 30 22:56:30.540409 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:30.540390 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f\": container with ID starting with 72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f not found: ID does not exist" containerID="72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f" Oct 30 22:56:30.540471 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.540417 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f"} err="failed to get container status \"72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f\": rpc error: code = NotFound desc = could not find container \"72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f\": container with ID starting with 72e1d35aa5910c08e70dc564e17ac47bb94c15e200ed09da70d5f50b2cb5c13f not found: ID does not exist" Oct 30 22:56:30.734873 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.734849 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9"] Oct 30 22:56:30.739349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.739325 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b392d-7f8dbbd897-z28h9"] Oct 30 22:56:30.869173 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:30.869100 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c397a2bb-d114-4394-ab83-5b3c0ba77994" path="/var/lib/kubelet/pods/c397a2bb-d114-4394-ab83-5b3c0ba77994/volumes" Oct 30 22:56:32.332416 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:32.332347 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerName="payload-processing" containerID="cri-o://88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c" gracePeriod=25 Oct 30 22:56:32.442434 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:32.442398 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerName="payload-pre-processing" containerID="cri-o://6b514eb1950b897db9bbaed005f0098ba083971417fd45294751e231357d7d42" gracePeriod=25 Oct 30 22:56:33.083805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:33.083770 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv"] Oct 30 22:56:33.084031 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:33.083996 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" containerID="cri-o://4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6" gracePeriod=30 Oct 30 22:56:37.013916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:37.013883 5607 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:56:37.014342 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:37.013972 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:56:37.902279 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:37.902241 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:38.356361 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.356338 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 22:56:38.446183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446153 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446198 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446226 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446276 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446328 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446326 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446377 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446415 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446491 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446517 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:38.446584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446571 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy\") pod \"60599210-aca4-4ef9-947a-3b21c31b55d9\" (UID: \"60599210-aca4-4ef9-947a-3b21c31b55d9\") " Oct 30 22:56:38.446584 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446570 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:38.446918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446788 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:38.446918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446840 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:38.446918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446862 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:38.446918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446888 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.446918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446909 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.446918 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.446923 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.448880 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.448858 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "kube-api-access-7vxmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:38.449004 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.448965 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:38.449302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.449281 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:38.449302 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.449284 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo" pod "60599210-aca4-4ef9-947a-3b21c31b55d9" (UID: "60599210-aca4-4ef9-947a-3b21c31b55d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:56:38.547481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.547454 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.547481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.547475 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.547481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.547485 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599210-aca4-4ef9-947a-3b21c31b55d9-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.547729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.547495 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599210-aca4-4ef9-947a-3b21c31b55d9-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.547729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.547504 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599210-aca4-4ef9-947a-3b21c31b55d9-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.547729 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.547513 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vxmv\" (UniqueName: \"kubernetes.io/projected/60599210-aca4-4ef9-947a-3b21c31b55d9-kube-api-access-7vxmv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:38.574474 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.574446 5607 generic.go:356] "Generic (PLEG): container finished" podID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerID="4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6" exitCode=0 Oct 30 22:56:38.574594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.574492 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" Oct 30 22:56:38.574594 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.574536 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" event={"ID":"60599210-aca4-4ef9-947a-3b21c31b55d9","Type":"ContainerDied","Data":"4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6"} Oct 30 22:56:38.574681 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.574644 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv" event={"ID":"60599210-aca4-4ef9-947a-3b21c31b55d9","Type":"ContainerDied","Data":"ce362eeda1e787f6c5cfdab0f5f2d201971363e5269b1cab73454b7b2bf039b3"} Oct 30 22:56:38.574716 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.574680 5607 scope.go:122] "RemoveContainer" containerID="4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6" Oct 30 22:56:38.584823 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.584806 5607 scope.go:122] "RemoveContainer" containerID="3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7" Oct 30 22:56:38.593615 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.593599 5607 scope.go:122] "RemoveContainer" containerID="4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6" Oct 30 22:56:38.593890 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:38.593868 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6\": container with ID starting with 4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6 not found: ID does not exist" containerID="4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6" Oct 30 22:56:38.593988 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.593900 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6"} err="failed to get container status \"4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6\": rpc error: code = NotFound desc = could not find container \"4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6\": container with ID starting with 4a38653b883113c43ee5ca13bf1bfc5e6d6f5d4eec3081dfee0030e0155c06d6 not found: ID does not exist" Oct 30 22:56:38.593988 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.593922 5607 scope.go:122] "RemoveContainer" containerID="3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7" Oct 30 22:56:38.594220 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:38.594201 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7\": container with ID starting with 3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7 not found: ID does not exist" containerID="3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7" Oct 30 22:56:38.594263 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.594227 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7"} err="failed to get container status \"3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7\": rpc error: code = NotFound desc = could not find container \"3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7\": container with ID starting with 3ebd4ae0b8fef985a6237f56cde293012e3effca1c8667b7aeadc41eadf634e7 not found: ID does not exist" Oct 30 22:56:38.744098 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.744046 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv"] Oct 30 22:56:38.755288 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.755263 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-d42dv"] Oct 30 22:56:38.869398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:38.869373 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" path="/var/lib/kubelet/pods/60599210-aca4-4ef9-947a-3b21c31b55d9/volumes" Oct 30 22:56:39.916742 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:39.916704 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:47.901646 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:47.901606 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:49.915519 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:49.915475 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:56:52.599360 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.599300 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:56:52.653462 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.653419 5607 generic.go:356] "Generic (PLEG): container finished" podID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerID="6b514eb1950b897db9bbaed005f0098ba083971417fd45294751e231357d7d42" exitCode=0 Oct 30 22:56:52.653637 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.653484 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" event={"ID":"78ca2c17-2aa9-4b52-b9da-356d099811b9","Type":"ContainerDied","Data":"6b514eb1950b897db9bbaed005f0098ba083971417fd45294751e231357d7d42"} Oct 30 22:56:52.655094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.655071 5607 generic.go:356] "Generic (PLEG): container finished" podID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerID="88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c" exitCode=0 Oct 30 22:56:52.655217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.655129 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" Oct 30 22:56:52.655217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.655154 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" event={"ID":"571973ac-dd21-4647-8eda-a9d56f5e9d73","Type":"ContainerDied","Data":"88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c"} Oct 30 22:56:52.655217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.655205 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st" event={"ID":"571973ac-dd21-4647-8eda-a9d56f5e9d73","Type":"ContainerDied","Data":"3fee49dd7c7088798ab7c8adc10d60ac530934abae0ac677ec408bed658df000"} Oct 30 22:56:52.655412 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.655236 5607 scope.go:122] "RemoveContainer" containerID="88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c" Oct 30 22:56:52.664274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.664245 5607 scope.go:122] "RemoveContainer" containerID="88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c" Oct 30 22:56:52.664568 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:52.664544 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c\": container with ID starting with 88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c not found: ID does not exist" containerID="88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c" Oct 30 22:56:52.664625 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.664581 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c"} err="failed to get container status \"88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c\": rpc error: code = NotFound desc = could not find container \"88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c\": container with ID starting with 88eee63203881c3b790d63504f9b6518fc276b6781d10aaa74d18eceb1eaca6c not found: ID does not exist" Oct 30 22:56:52.668938 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.668915 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume\") pod \"571973ac-dd21-4647-8eda-a9d56f5e9d73\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " Oct 30 22:56:52.669033 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.669009 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx\" (UniqueName: \"kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx\") pod \"571973ac-dd21-4647-8eda-a9d56f5e9d73\" (UID: \"571973ac-dd21-4647-8eda-a9d56f5e9d73\") " Oct 30 22:56:52.669441 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.669418 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume" pod "571973ac-dd21-4647-8eda-a9d56f5e9d73" (UID: "571973ac-dd21-4647-8eda-a9d56f5e9d73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:52.669530 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.669515 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571973ac-dd21-4647-8eda-a9d56f5e9d73-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:52.671229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.671207 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx" pod "571973ac-dd21-4647-8eda-a9d56f5e9d73" (UID: "571973ac-dd21-4647-8eda-a9d56f5e9d73"). InnerVolumeSpecName "kube-api-access-rrzhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:52.690332 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.690312 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:56:52.770728 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.770704 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462\" (UniqueName: \"kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462\") pod \"78ca2c17-2aa9-4b52-b9da-356d099811b9\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " Oct 30 22:56:52.770896 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.770769 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume\") pod \"78ca2c17-2aa9-4b52-b9da-356d099811b9\" (UID: \"78ca2c17-2aa9-4b52-b9da-356d099811b9\") " Oct 30 22:56:52.771241 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.771216 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzhx\" (UniqueName: \"kubernetes.io/projected/571973ac-dd21-4647-8eda-a9d56f5e9d73-kube-api-access-rrzhx\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:52.771389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.771226 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume" pod "78ca2c17-2aa9-4b52-b9da-356d099811b9" (UID: "78ca2c17-2aa9-4b52-b9da-356d099811b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:52.773155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.773111 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462" pod "78ca2c17-2aa9-4b52-b9da-356d099811b9" (UID: "78ca2c17-2aa9-4b52-b9da-356d099811b9"). InnerVolumeSpecName "kube-api-access-bx462". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:52.863332 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:56:52.863257 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:56:52.872059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.872031 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ca2c17-2aa9-4b52-b9da-356d099811b9-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:52.872059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:52.872058 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx462\" (UniqueName: \"kubernetes.io/projected/78ca2c17-2aa9-4b52-b9da-356d099811b9-kube-api-access-bx462\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:53.071651 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.071618 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st"] Oct 30 22:56:53.076387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.076360 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b392d-7d45498b8c-2d5st"] Oct 30 22:56:53.661740 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.661651 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" Oct 30 22:56:53.662289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.661650 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx" event={"ID":"78ca2c17-2aa9-4b52-b9da-356d099811b9","Type":"ContainerDied","Data":"1cb0e232335ff95720c8fa9715d63e4c6c2102d4b197bf891879a7fed19cb58f"} Oct 30 22:56:53.662289 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.661806 5607 scope.go:122] "RemoveContainer" containerID="6b514eb1950b897db9bbaed005f0098ba083971417fd45294751e231357d7d42" Oct 30 22:56:53.830567 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.830527 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx"] Oct 30 22:56:53.835481 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:53.835460 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b392d-5f56c784d7-2jnsx"] Oct 30 22:56:54.869114 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:54.869082 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" path="/var/lib/kubelet/pods/571973ac-dd21-4647-8eda-a9d56f5e9d73/volumes" Oct 30 22:56:54.869517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:56:54.869465 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" path="/var/lib/kubelet/pods/78ca2c17-2aa9-4b52-b9da-356d099811b9/volumes" Oct 30 22:57:01.488306 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:01.488275 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:57:01.496379 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:01.496357 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2s925_18cf0a1a-efb8-4aef-ac2a-16e294018ccf/ovn-acl-logging/0.log" Oct 30 22:57:06.015260 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:06.015217 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc"] Oct 30 22:57:06.126475 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:06.126437 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd"] Oct 30 22:57:11.016075 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:11.016029 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerName="payload-processing" containerID="cri-o://728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3" gracePeriod=25 Oct 30 22:57:11.127507 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:11.127465 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" podUID="eb0f2b62-a071-4893-9003-da224c78556c" containerName="payload-pre-processing" containerID="cri-o://eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e" gracePeriod=25 Oct 30 22:57:11.961660 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:11.961620 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:12.968915 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:12.968879 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" podUID="eb0f2b62-a071-4893-9003-da224c78556c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:13.433009 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.432906 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn"] Oct 30 22:57:13.433166 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.433129 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" podUID="0721b687-47df-4042-b54f-50f781677f0d" containerName="maas-api" containerID="cri-o://c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5" gracePeriod=30 Oct 30 22:57:13.684079 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.684027 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:57:13.764439 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.764409 5607 generic.go:356] "Generic (PLEG): container finished" podID="0721b687-47df-4042-b54f-50f781677f0d" containerID="c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5" exitCode=0 Oct 30 22:57:13.764583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.764455 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" Oct 30 22:57:13.764583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.764498 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" event={"ID":"0721b687-47df-4042-b54f-50f781677f0d","Type":"ContainerDied","Data":"c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5"} Oct 30 22:57:13.764583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.764550 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn" event={"ID":"0721b687-47df-4042-b54f-50f781677f0d","Type":"ContainerDied","Data":"e2cd5cf488e26dd559de91fadfadb4c10f4f1ba3fd8a73d139848180af94c7cb"} Oct 30 22:57:13.764583 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.764572 5607 scope.go:122] "RemoveContainer" containerID="c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5" Oct 30 22:57:13.773422 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.773407 5607 scope.go:122] "RemoveContainer" containerID="c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5" Oct 30 22:57:13.773692 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:13.773675 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5\": container with ID starting with c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5 not found: ID does not exist" containerID="c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5" Oct 30 22:57:13.773768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.773699 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5"} err="failed to get container status \"c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5\": rpc error: code = NotFound desc = could not find container \"c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5\": container with ID starting with c946dbe088fdd1f87a32f0b31b31da7d18741f30b27fc7e3901f57d56e70b6f5 not found: ID does not exist" Oct 30 22:57:13.859012 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.858939 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") pod \"0721b687-47df-4042-b54f-50f781677f0d\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " Oct 30 22:57:13.859214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.859062 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs\") pod \"0721b687-47df-4042-b54f-50f781677f0d\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " Oct 30 22:57:13.859214 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.859128 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5\" (UniqueName: \"kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5\") pod \"0721b687-47df-4042-b54f-50f781677f0d\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " Oct 30 22:57:13.859343 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.859239 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca\") pod \"0721b687-47df-4042-b54f-50f781677f0d\" (UID: \"0721b687-47df-4042-b54f-50f781677f0d\") " Oct 30 22:57:13.859670 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.859636 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca" pod "0721b687-47df-4042-b54f-50f781677f0d" (UID: "0721b687-47df-4042-b54f-50f781677f0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:13.861389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.861359 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs" pod "0721b687-47df-4042-b54f-50f781677f0d" (UID: "0721b687-47df-4042-b54f-50f781677f0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:13.861480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.861407 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls" pod "0721b687-47df-4042-b54f-50f781677f0d" (UID: "0721b687-47df-4042-b54f-50f781677f0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:13.861480 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.861445 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5" pod "0721b687-47df-4042-b54f-50f781677f0d" (UID: "0721b687-47df-4042-b54f-50f781677f0d"). InnerVolumeSpecName "kube-api-access-62xv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:13.960665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.960640 5607 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0721b687-47df-4042-b54f-50f781677f0d-openshift-service-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.960665 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.960666 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-maas-api-tls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.960805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.960675 5607 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0721b687-47df-4042-b54f-50f781677f0d-metrics-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:13.960805 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:13.960685 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62xv5\" (UniqueName: \"kubernetes.io/projected/0721b687-47df-4042-b54f-50f781677f0d-kube-api-access-62xv5\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:14.188001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:14.187946 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn"] Oct 30 22:57:14.192425 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:14.192406 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-74bb81-59fd956c8c-m8tnn"] Oct 30 22:57:14.868626 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:14.868591 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0721b687-47df-4042-b54f-50f781677f0d" path="/var/lib/kubelet/pods/0721b687-47df-4042-b54f-50f781677f0d/volumes" Oct 30 22:57:21.962047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:21.962009 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:22.862705 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:22.862662 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:57:22.968788 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:22.968758 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" podUID="eb0f2b62-a071-4893-9003-da224c78556c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:23.420520 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420442 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m"] Oct 30 22:57:23.420921 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420907 5607 cpu_manager.go:414] "containerMap: removing container" podUID="0721b687-47df-4042-b54f-50f781677f0d" containerName="maas-api" Oct 30 22:57:23.420992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420924 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0721b687-47df-4042-b54f-50f781677f0d" containerName="maas-api" Oct 30 22:57:23.420992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420940 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c397a2bb-d114-4394-ab83-5b3c0ba77994" containerName="maas-api" Oct 30 22:57:23.420992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420945 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c397a2bb-d114-4394-ab83-5b3c0ba77994" containerName="maas-api" Oct 30 22:57:23.420992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420980 5607 cpu_manager.go:414] "containerMap: removing container" podUID="37711f92-7c68-4c03-97c8-c86d2e012007" containerName="payload-processing" Oct 30 22:57:23.420992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420985 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37711f92-7c68-4c03-97c8-c86d2e012007" containerName="payload-processing" Oct 30 22:57:23.420992 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420993 5607 cpu_manager.go:414] "containerMap: removing container" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.420997 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421005 5607 cpu_manager.go:414] "containerMap: removing container" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421009 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421016 5607 cpu_manager.go:414] "containerMap: removing container" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421020 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421027 5607 cpu_manager.go:414] "containerMap: removing container" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421034 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421043 5607 cpu_manager.go:414] "containerMap: removing container" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421048 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421056 5607 cpu_manager.go:414] "containerMap: removing container" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421060 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerName="payload-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421068 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" containerName="maas-api" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421072 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" containerName="maas-api" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421079 5607 cpu_manager.go:414] "containerMap: removing container" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421083 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421094 5607 cpu_manager.go:414] "containerMap: removing container" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421098 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421105 5607 cpu_manager.go:414] "containerMap: removing container" podUID="eec8660c-601d-4bdc-a880-4e6d9e2c1415" containerName="maas-api" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421111 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec8660c-601d-4bdc-a880-4e6d9e2c1415" containerName="maas-api" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421126 5607 cpu_manager.go:414] "containerMap: removing container" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421132 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421141 5607 cpu_manager.go:414] "containerMap: removing container" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421147 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421155 5607 cpu_manager.go:414] "containerMap: removing container" podUID="375990b8-fabe-4822-881d-d776174dd6ec" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421159 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375990b8-fabe-4822-881d-d776174dd6ec" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421166 5607 cpu_manager.go:414] "containerMap: removing container" podUID="69f559a6-a076-4147-8069-694ec06a8575" containerName="maas-api" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421171 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f559a6-a076-4147-8069-694ec06a8575" containerName="maas-api" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421178 5607 cpu_manager.go:414] "containerMap: removing container" podUID="8e411daa-bada-4599-9c96-38e1dc7d1bac" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421182 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e411daa-bada-4599-9c96-38e1dc7d1bac" containerName="payload-pre-processing" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421189 5607 cpu_manager.go:414] "containerMap: removing container" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421196 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" Oct 30 22:57:23.421190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421205 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421210 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421216 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a4c5d315-965e-4315-a243-0af5c701cfd2" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421221 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c5d315-965e-4315-a243-0af5c701cfd2" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421229 5607 cpu_manager.go:414] "containerMap: removing container" podUID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421233 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421248 5607 cpu_manager.go:414] "containerMap: removing container" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421252 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421258 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421262 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421276 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2f0b9648-ca39-4146-a501-040489a211c2" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421283 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0b9648-ca39-4146-a501-040489a211c2" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421288 5607 cpu_manager.go:414] "containerMap: removing container" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421292 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421300 5607 cpu_manager.go:414] "containerMap: removing container" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421304 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421310 5607 cpu_manager.go:414] "containerMap: removing container" podUID="3c486f28-0c4c-4065-910a-3d590c82003d" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421314 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c486f28-0c4c-4065-910a-3d590c82003d" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421321 5607 cpu_manager.go:414] "containerMap: removing container" podUID="09f8a61b-d445-4b2d-b66c-e37b18c93114" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421325 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f8a61b-d445-4b2d-b66c-e37b18c93114" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421332 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2216473b-6bc3-4d55-b118-9bc2de085859" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421336 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2216473b-6bc3-4d55-b118-9bc2de085859" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421345 5607 cpu_manager.go:414] "containerMap: removing container" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421352 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421362 5607 cpu_manager.go:414] "containerMap: removing container" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421367 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421372 5607 cpu_manager.go:414] "containerMap: removing container" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421377 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421442 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="571973ac-dd21-4647-8eda-a9d56f5e9d73" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421452 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c486f28-0c4c-4065-910a-3d590c82003d" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421458 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f8a61b-d445-4b2d-b66c-e37b18c93114" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421463 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2216473b-6bc3-4d55-b118-9bc2de085859" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421468 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1afb645-4dd0-4d16-8899-e65634921b01" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421475 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b73df7-7c61-4a96-8813-cb5b3548b31c" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421480 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59dd0a1-3355-4607-aecc-d959da666b1a" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421486 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="0721b687-47df-4042-b54f-50f781677f0d" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421492 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c397a2bb-d114-4394-ab83-5b3c0ba77994" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421497 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="37711f92-7c68-4c03-97c8-c86d2e012007" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421503 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="549f9160-eeb0-40e9-81f8-dfeecc639d62" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421510 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="7899e2f7-a654-490b-aa02-af4124d6b43b" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421520 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b3ab7b-6fa5-479a-8dfb-b4f1b9e399db" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421526 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa9bead-f06b-4467-8d59-fa25bf5354ca" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421531 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="810d67a6-711d-44c7-a72f-5b6e1b405026" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421537 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="45326de7-55f6-4593-9ae2-827a3f87eefb" containerName="payload-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421543 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb8a7ea-b6b9-4aaa-b0af-aaa2960ba88d" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421548 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="450a2dde-cc87-40f6-bae8-ce0d32dd1428" containerName="main" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421554 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a7fe83-aac9-44eb-847a-d3246cd0a1de" containerName="main" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421559 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec8660c-601d-4bdc-a880-4e6d9e2c1415" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421564 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ca2c17-2aa9-4b52-b9da-356d099811b9" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421570 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd9a3b5-8766-45f2-a471-68f8443cc2f6" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421575 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="375990b8-fabe-4822-881d-d776174dd6ec" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421580 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f559a6-a076-4147-8069-694ec06a8575" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421585 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e411daa-bada-4599-9c96-38e1dc7d1bac" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421593 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421601 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3dd339-b978-4b7a-998b-587e00c82bca" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421606 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c5d315-965e-4315-a243-0af5c701cfd2" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421610 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e54bdf-abbb-4a76-ae6e-c98d237359d1" containerName="maas-api" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421615 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6e63ab-61f5-4da2-80e1-eac2983b35fe" containerName="main" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421619 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a5bbc0-5baf-43db-a70a-ea81a102c270" containerName="istio-proxy" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421623 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0b9648-ca39-4146-a501-040489a211c2" containerName="payload-pre-processing" Oct 30 22:57:23.422264 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.421628 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="60599210-aca4-4ef9-947a-3b21c31b55d9" containerName="istio-proxy" Oct 30 22:57:23.426417 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.426401 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.484886 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.484857 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m"] Oct 30 22:57:23.533732 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.533705 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:57:23.533879 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.533743 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:57:23.549132 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.549104 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.549269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.549226 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.549269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.549263 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lt8\" (UniqueName: \"kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.649773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.649741 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.649941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.649781 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lt8\" (UniqueName: \"kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.649941 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.649860 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.650398 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.650380 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.652397 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.652374 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.712389 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.712359 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lt8\" (UniqueName: \"kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8\") pod \"maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:23.739912 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:23.739884 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:24.106388 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:57:24.106349 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a589e9_7a25_4f16_ab8a_a960ef239e3c.slice/crio-ea7a7c90172e17e1bd74ed2a6ee2e0e6c6c7ee6b6ffd169c276b3fe4cbbbbd5f WatchSource:0}: Error finding container ea7a7c90172e17e1bd74ed2a6ee2e0e6c6c7ee6b6ffd169c276b3fe4cbbbbd5f: Status 404 returned error can't find the container with id ea7a7c90172e17e1bd74ed2a6ee2e0e6c6c7ee6b6ffd169c276b3fe4cbbbbd5f Oct 30 22:57:24.108591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:24.108572 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m"] Oct 30 22:57:24.819699 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:24.819665 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" event={"ID":"e0a589e9-7a25-4f16-ab8a-a960ef239e3c","Type":"ContainerStarted","Data":"3ea4f26c936da36f3d2009be0be2aa275e7b39104356d8b92b66489c59898304"} Oct 30 22:57:24.819924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:24.819725 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" event={"ID":"e0a589e9-7a25-4f16-ab8a-a960ef239e3c","Type":"ContainerStarted","Data":"ea7a7c90172e17e1bd74ed2a6ee2e0e6c6c7ee6b6ffd169c276b3fe4cbbbbd5f"} Oct 30 22:57:24.939870 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:24.939815 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" podStartSLOduration=1.939801428 podStartE2EDuration="1.939801428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:24.938053708 +0000 UTC m=+4224.338609832" watchObservedRunningTime="2026-10-30 22:57:24.939801428 +0000 UTC m=+4224.340357589" Oct 30 22:57:25.826700 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:25.826665 5607 generic.go:356] "Generic (PLEG): container finished" podID="e0a589e9-7a25-4f16-ab8a-a960ef239e3c" containerID="3ea4f26c936da36f3d2009be0be2aa275e7b39104356d8b92b66489c59898304" exitCode=0 Oct 30 22:57:25.827134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:25.826742 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" event={"ID":"e0a589e9-7a25-4f16-ab8a-a960ef239e3c","Type":"ContainerDied","Data":"3ea4f26c936da36f3d2009be0be2aa275e7b39104356d8b92b66489c59898304"} Oct 30 22:57:26.972858 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.972831 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:26.979014 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.978995 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca\") pod \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " Oct 30 22:57:26.979111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.979045 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca\") pod \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " Oct 30 22:57:26.979111 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.979103 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8\" (UniqueName: \"kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8\") pod \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\" (UID: \"e0a589e9-7a25-4f16-ab8a-a960ef239e3c\") " Oct 30 22:57:26.979388 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.979365 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca" pod "e0a589e9-7a25-4f16-ab8a-a960ef239e3c" (UID: "e0a589e9-7a25-4f16-ab8a-a960ef239e3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:26.981170 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.981144 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca" pod "e0a589e9-7a25-4f16-ab8a-a960ef239e3c" (UID: "e0a589e9-7a25-4f16-ab8a-a960ef239e3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:26.981274 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:26.981210 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8" pod "e0a589e9-7a25-4f16-ab8a-a960ef239e3c" (UID: "e0a589e9-7a25-4f16-ab8a-a960ef239e3c"). InnerVolumeSpecName "kube-api-access-q8lt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:27.080793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.080769 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:27.080793 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.080790 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:27.080925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.080800 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8lt8\" (UniqueName: \"kubernetes.io/projected/e0a589e9-7a25-4f16-ab8a-a960ef239e3c-kube-api-access-q8lt8\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:27.837399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.837359 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" event={"ID":"e0a589e9-7a25-4f16-ab8a-a960ef239e3c","Type":"ContainerDied","Data":"ea7a7c90172e17e1bd74ed2a6ee2e0e6c6c7ee6b6ffd169c276b3fe4cbbbbd5f"} Oct 30 22:57:27.837399 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.837396 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m" Oct 30 22:57:27.837607 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.837419 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea7a7c90172e17e1bd74ed2a6ee2e0e6c6c7ee6b6ffd169c276b3fe4cbbbbd5f" Oct 30 22:57:27.952040 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.952013 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5"] Oct 30 22:57:27.953435 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:27.953412 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6"] Oct 30 22:57:31.273450 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.273392 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:57:31.318571 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.318540 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq\" (UniqueName: \"kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq\") pod \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " Oct 30 22:57:31.318741 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.318683 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume\") pod \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\" (UID: \"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1\") " Oct 30 22:57:31.319063 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.319038 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume" pod "52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" (UID: "52b02ea1-6c5f-44dc-89de-1d7e2241c7d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:31.321502 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.321459 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq" pod "52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" (UID: "52b02ea1-6c5f-44dc-89de-1d7e2241c7d1"). InnerVolumeSpecName "kube-api-access-jjrsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:31.368760 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.368735 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:57:31.419650 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.419622 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25\" (UniqueName: \"kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25\") pod \"eb0f2b62-a071-4893-9003-da224c78556c\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " Oct 30 22:57:31.419835 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.419660 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume\") pod \"eb0f2b62-a071-4893-9003-da224c78556c\" (UID: \"eb0f2b62-a071-4893-9003-da224c78556c\") " Oct 30 22:57:31.420089 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.420069 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjrsq\" (UniqueName: \"kubernetes.io/projected/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-kube-api-access-jjrsq\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:31.420163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.420094 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:31.420163 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.420134 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume" pod "eb0f2b62-a071-4893-9003-da224c78556c" (UID: "eb0f2b62-a071-4893-9003-da224c78556c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:31.421947 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.421932 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25" pod "eb0f2b62-a071-4893-9003-da224c78556c" (UID: "eb0f2b62-a071-4893-9003-da224c78556c"). InnerVolumeSpecName "kube-api-access-gsb25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:31.521345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.521308 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsb25\" (UniqueName: \"kubernetes.io/projected/eb0f2b62-a071-4893-9003-da224c78556c-kube-api-access-gsb25\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:31.521345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.521347 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb0f2b62-a071-4893-9003-da224c78556c-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:31.865363 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.865326 5607 generic.go:356] "Generic (PLEG): container finished" podID="eb0f2b62-a071-4893-9003-da224c78556c" containerID="eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e" exitCode=0 Oct 30 22:57:31.865569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.865371 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" Oct 30 22:57:31.865569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.865406 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" event={"ID":"eb0f2b62-a071-4893-9003-da224c78556c","Type":"ContainerDied","Data":"eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e"} Oct 30 22:57:31.865569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.865457 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd" event={"ID":"eb0f2b62-a071-4893-9003-da224c78556c","Type":"ContainerDied","Data":"d032873e30919dfa3f60cae0d8af87be0f2af6876dd4a03f5956f0f2b880f8d3"} Oct 30 22:57:31.865569 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.865481 5607 scope.go:122] "RemoveContainer" containerID="eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e" Oct 30 22:57:31.867144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.867078 5607 generic.go:356] "Generic (PLEG): container finished" podID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerID="728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3" exitCode=0 Oct 30 22:57:31.867144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.867111 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" event={"ID":"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1","Type":"ContainerDied","Data":"728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3"} Oct 30 22:57:31.867144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.867143 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" event={"ID":"52b02ea1-6c5f-44dc-89de-1d7e2241c7d1","Type":"ContainerDied","Data":"457d7cdc5b40f774d1e461374967762e1bf3d21e0cffb84ef091303535ee0ffe"} Oct 30 22:57:31.867144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.867150 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc" Oct 30 22:57:31.878687 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.878660 5607 scope.go:122] "RemoveContainer" containerID="eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e" Oct 30 22:57:31.879063 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:31.879042 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e\": container with ID starting with eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e not found: ID does not exist" containerID="eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e" Oct 30 22:57:31.879160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.879072 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e"} err="failed to get container status \"eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e\": rpc error: code = NotFound desc = could not find container \"eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e\": container with ID starting with eee0eab9273af06cb5c5f28d987370b7a29e29c71f23bf991fbdcc3e2294863e not found: ID does not exist" Oct 30 22:57:31.879160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.879092 5607 scope.go:122] "RemoveContainer" containerID="728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3" Oct 30 22:57:31.887775 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.887755 5607 scope.go:122] "RemoveContainer" containerID="728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3" Oct 30 22:57:31.888068 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:31.888046 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3\": container with ID starting with 728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3 not found: ID does not exist" containerID="728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3" Oct 30 22:57:31.888147 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:31.888075 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3"} err="failed to get container status \"728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3\": rpc error: code = NotFound desc = could not find container \"728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3\": container with ID starting with 728a22efd43c3bba46a0dd303b7df05a161b271e393860313fc5bc3f4faa02d3 not found: ID does not exist" Oct 30 22:57:32.033433 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:32.033399 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd"] Oct 30 22:57:32.038284 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:32.038263 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-74bb81-5f97856699-sphwd"] Oct 30 22:57:32.149818 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:32.149723 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc"] Oct 30 22:57:32.155825 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:32.155802 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-74bb81-66c8ffbfdc-6mqdc"] Oct 30 22:57:32.868977 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:32.868923 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" path="/var/lib/kubelet/pods/52b02ea1-6c5f-44dc-89de-1d7e2241c7d1/volumes" Oct 30 22:57:32.869533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:32.869514 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0f2b62-a071-4893-9003-da224c78556c" path="/var/lib/kubelet/pods/eb0f2b62-a071-4893-9003-da224c78556c/volumes" Oct 30 22:57:35.091674 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:35.091642 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j"] Oct 30 22:57:38.300251 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300209 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p"] Oct 30 22:57:38.300834 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300812 5607 cpu_manager.go:414] "containerMap: removing container" podUID="eb0f2b62-a071-4893-9003-da224c78556c" containerName="payload-pre-processing" Oct 30 22:57:38.300916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300836 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0f2b62-a071-4893-9003-da224c78556c" containerName="payload-pre-processing" Oct 30 22:57:38.300916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300890 5607 cpu_manager.go:414] "containerMap: removing container" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerName="payload-processing" Oct 30 22:57:38.300916 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300899 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerName="payload-processing" Oct 30 22:57:38.301134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300933 5607 cpu_manager.go:414] "containerMap: removing container" podUID="e0a589e9-7a25-4f16-ab8a-a960ef239e3c" containerName="revoke-keys" Oct 30 22:57:38.301134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.300942 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a589e9-7a25-4f16-ab8a-a960ef239e3c" containerName="revoke-keys" Oct 30 22:57:38.301134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.301071 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b02ea1-6c5f-44dc-89de-1d7e2241c7d1" containerName="payload-processing" Oct 30 22:57:38.301134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.301091 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a589e9-7a25-4f16-ab8a-a960ef239e3c" containerName="revoke-keys" Oct 30 22:57:38.301134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.301101 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0f2b62-a071-4893-9003-da224c78556c" containerName="payload-pre-processing" Oct 30 22:57:38.304134 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.304113 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.360689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.360664 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:57:38.386183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.386157 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjxd\" (UniqueName: \"kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.386335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.386197 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.386335 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.386234 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.411566 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.411536 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\"" Oct 30 22:57:38.413840 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.413818 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p"] Oct 30 22:57:38.487932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.487896 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjxd\" (UniqueName: \"kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.487932 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.487934 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.488242 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.488071 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.488724 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.488706 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.490630 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.490608 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.550717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.550641 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjxd\" (UniqueName: \"kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd\") pod \"maas-api-revoke-sub-91bcc99f-8fl7p\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.617351 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.617326 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:38.988269 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:38.988237 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p"] Oct 30 22:57:38.988426 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:57:38.988278 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509b73b9_8697_4da1_91a7_50cb970c8184.slice/crio-51a0937e4b78e569e2a5a45db9599ed8ebc6a50618487f384e7c9f9703dc7bd2 WatchSource:0}: Error finding container 51a0937e4b78e569e2a5a45db9599ed8ebc6a50618487f384e7c9f9703dc7bd2: Status 404 returned error can't find the container with id 51a0937e4b78e569e2a5a45db9599ed8ebc6a50618487f384e7c9f9703dc7bd2 Oct 30 22:57:39.907944 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:39.907913 5607 generic.go:356] "Generic (PLEG): container finished" podID="509b73b9-8697-4da1-91a7-50cb970c8184" containerID="cd239d7b45d8400a9f94a277e31562ab354f494cdb70da0a96fb54af6879afcb" exitCode=0 Oct 30 22:57:39.908330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:39.907986 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" event={"ID":"509b73b9-8697-4da1-91a7-50cb970c8184","Type":"ContainerDied","Data":"cd239d7b45d8400a9f94a277e31562ab354f494cdb70da0a96fb54af6879afcb"} Oct 30 22:57:39.908330 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:39.908053 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" event={"ID":"509b73b9-8697-4da1-91a7-50cb970c8184","Type":"ContainerStarted","Data":"51a0937e4b78e569e2a5a45db9599ed8ebc6a50618487f384e7c9f9703dc7bd2"} Oct 30 22:57:40.093096 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:40.093052 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerName="payload-pre-processing" containerID="cri-o://51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44" gracePeriod=25 Oct 30 22:57:41.051653 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.051625 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:41.115517 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.115490 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd\" (UniqueName: \"kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd\") pod \"509b73b9-8697-4da1-91a7-50cb970c8184\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " Oct 30 22:57:41.115702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.115538 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca\") pod \"509b73b9-8697-4da1-91a7-50cb970c8184\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " Oct 30 22:57:41.115702 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.115619 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca\") pod \"509b73b9-8697-4da1-91a7-50cb970c8184\" (UID: \"509b73b9-8697-4da1-91a7-50cb970c8184\") " Oct 30 22:57:41.116046 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.116018 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca" pod "509b73b9-8697-4da1-91a7-50cb970c8184" (UID: "509b73b9-8697-4da1-91a7-50cb970c8184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:57:41.117833 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.117812 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca" pod "509b73b9-8697-4da1-91a7-50cb970c8184" (UID: "509b73b9-8697-4da1-91a7-50cb970c8184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:41.117935 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.117856 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd" pod "509b73b9-8697-4da1-91a7-50cb970c8184" (UID: "509b73b9-8697-4da1-91a7-50cb970c8184"). InnerVolumeSpecName "kube-api-access-vtjxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:41.216757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.216731 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjxd\" (UniqueName: \"kubernetes.io/projected/509b73b9-8697-4da1-91a7-50cb970c8184-kube-api-access-vtjxd\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:41.216757 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.216757 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-xks-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:41.216984 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.216767 5607 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/509b73b9-8697-4da1-91a7-50cb970c8184-maas-api-openshift-ca\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:41.918320 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.918288 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" Oct 30 22:57:41.918526 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.918317 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91bcc99f-8fl7p" event={"ID":"509b73b9-8697-4da1-91a7-50cb970c8184","Type":"ContainerDied","Data":"51a0937e4b78e569e2a5a45db9599ed8ebc6a50618487f384e7c9f9703dc7bd2"} Oct 30 22:57:41.918526 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:41.918362 5607 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a0937e4b78e569e2a5a45db9599ed8ebc6a50618487f384e7c9f9703dc7bd2" Oct 30 22:57:42.972869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:42.972829 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" containerID="cri-o://156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e" gracePeriod=45 Oct 30 22:57:42.972869 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:42.972874 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" containerID="cri-o://85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171" gracePeriod=45 Oct 30 22:57:43.003002 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:43.002978 5607 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5591863e_ed98_405d_b7ce_7c0c9eeb099f.slice/crio-156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:57:43.081900 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.081870 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": dial tcp 10.133.0.97:8000: connect: connection refused" Oct 30 22:57:43.231427 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.231366 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:57:43.263794 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.263770 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:57:43.335490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335465 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home\") pod \"dfaba7da-6325-41c6-87da-db337fffd973\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " Oct 30 22:57:43.335647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335528 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir\") pod \"dfaba7da-6325-41c6-87da-db337fffd973\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " Oct 30 22:57:43.335647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335572 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache\") pod \"dfaba7da-6325-41c6-87da-db337fffd973\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " Oct 30 22:57:43.335647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335614 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs\") pod \"dfaba7da-6325-41c6-87da-db337fffd973\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " Oct 30 22:57:43.335647 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335633 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv\" (UniqueName: \"kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv\") pod \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335679 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir\") pod \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335706 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache\") pod \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335732 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home\") pod \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335770 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm\") pod \"dfaba7da-6325-41c6-87da-db337fffd973\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335756 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home" pod "dfaba7da-6325-41c6-87da-db337fffd973" (UID: "dfaba7da-6325-41c6-87da-db337fffd973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335811 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm\") pod \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " Oct 30 22:57:43.335893 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335880 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs\" (UniqueName: \"kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs\") pod \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\" (UID: \"5591863e-ed98-405d-b7ce-7c0c9eeb099f\") " Oct 30 22:57:43.336303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335942 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6\" (UniqueName: \"kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6\") pod \"dfaba7da-6325-41c6-87da-db337fffd973\" (UID: \"dfaba7da-6325-41c6-87da-db337fffd973\") " Oct 30 22:57:43.336303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335840 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir" pod "dfaba7da-6325-41c6-87da-db337fffd973" (UID: "dfaba7da-6325-41c6-87da-db337fffd973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.336303 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.336222 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home" pod "5591863e-ed98-405d-b7ce-7c0c9eeb099f" (UID: "5591863e-ed98-405d-b7ce-7c0c9eeb099f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.336455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.336368 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.336455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.336385 5607 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-home\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.336455 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.336400 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.336797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335932 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache" pod "dfaba7da-6325-41c6-87da-db337fffd973" (UID: "dfaba7da-6325-41c6-87da-db337fffd973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.336797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.335999 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir" pod "5591863e-ed98-405d-b7ce-7c0c9eeb099f" (UID: "5591863e-ed98-405d-b7ce-7c0c9eeb099f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.336797 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.336172 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache" pod "5591863e-ed98-405d-b7ce-7c0c9eeb099f" (UID: "5591863e-ed98-405d-b7ce-7c0c9eeb099f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.339614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.339545 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs" pod "dfaba7da-6325-41c6-87da-db337fffd973" (UID: "dfaba7da-6325-41c6-87da-db337fffd973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:43.339614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.339532 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm" pod "dfaba7da-6325-41c6-87da-db337fffd973" (UID: "dfaba7da-6325-41c6-87da-db337fffd973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.339614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.339563 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv" pod "5591863e-ed98-405d-b7ce-7c0c9eeb099f" (UID: "5591863e-ed98-405d-b7ce-7c0c9eeb099f"). InnerVolumeSpecName "kube-api-access-plndv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:43.339614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.339573 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm" pod "5591863e-ed98-405d-b7ce-7c0c9eeb099f" (UID: "5591863e-ed98-405d-b7ce-7c0c9eeb099f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:57:43.339614 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.339545 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6" pod "dfaba7da-6325-41c6-87da-db337fffd973" (UID: "dfaba7da-6325-41c6-87da-db337fffd973"). InnerVolumeSpecName "kube-api-access-kq4j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:57:43.340035 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.339663 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs" pod "5591863e-ed98-405d-b7ce-7c0c9eeb099f" (UID: "5591863e-ed98-405d-b7ce-7c0c9eeb099f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:57:43.437813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437783 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.437813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437810 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfaba7da-6325-41c6-87da-db337fffd973-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.437813 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437820 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plndv\" (UniqueName: \"kubernetes.io/projected/5591863e-ed98-405d-b7ce-7c0c9eeb099f-kube-api-access-plndv\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.438047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437829 5607 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tmp-dir\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.438047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437837 5607 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-model-cache\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.438047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437845 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfaba7da-6325-41c6-87da-db337fffd973-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.438047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437852 5607 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5591863e-ed98-405d-b7ce-7c0c9eeb099f-dshm\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.438047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437860 5607 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5591863e-ed98-405d-b7ce-7c0c9eeb099f-tls-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.438047 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.437867 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq4j6\" (UniqueName: \"kubernetes.io/projected/dfaba7da-6325-41c6-87da-db337fffd973-kube-api-access-kq4j6\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:57:43.929219 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.929189 5607 generic.go:356] "Generic (PLEG): container finished" podID="dfaba7da-6325-41c6-87da-db337fffd973" containerID="85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171" exitCode=0 Oct 30 22:57:43.929484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.929234 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" Oct 30 22:57:43.929484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.929279 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" event={"ID":"dfaba7da-6325-41c6-87da-db337fffd973","Type":"ContainerDied","Data":"85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171"} Oct 30 22:57:43.929484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.929322 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" event={"ID":"dfaba7da-6325-41c6-87da-db337fffd973","Type":"ContainerDied","Data":"fc247f9c7a445e78b5cedc6f773b6933e3adc5b269dff4ae312f80a4de124670"} Oct 30 22:57:43.929484 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.929343 5607 scope.go:122] "RemoveContainer" containerID="85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171" Oct 30 22:57:43.930608 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.930586 5607 generic.go:356] "Generic (PLEG): container finished" podID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerID="156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e" exitCode=0 Oct 30 22:57:43.930735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.930632 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" Oct 30 22:57:43.930735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.930640 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" event={"ID":"5591863e-ed98-405d-b7ce-7c0c9eeb099f","Type":"ContainerDied","Data":"156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e"} Oct 30 22:57:43.930735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.930666 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5" event={"ID":"5591863e-ed98-405d-b7ce-7c0c9eeb099f","Type":"ContainerDied","Data":"9dbd9563f29e1fc53d4944f6e1d14c7febe3263bd9bb22a7309cf14bb1727792"} Oct 30 22:57:43.939221 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.939201 5607 scope.go:122] "RemoveContainer" containerID="85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171" Oct 30 22:57:43.939519 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:43.939493 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171\": container with ID starting with 85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171 not found: ID does not exist" containerID="85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171" Oct 30 22:57:43.939600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.939529 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171"} err="failed to get container status \"85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171\": rpc error: code = NotFound desc = could not find container \"85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171\": container with ID starting with 85adccdbba9424139e9251805057fccc3f01672ef9536cad850a0f0b62ebb171 not found: ID does not exist" Oct 30 22:57:43.939600 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.939547 5607 scope.go:122] "RemoveContainer" containerID="156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e" Oct 30 22:57:43.947464 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.947444 5607 scope.go:122] "RemoveContainer" containerID="156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e" Oct 30 22:57:43.947751 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:43.947732 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e\": container with ID starting with 156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e not found: ID does not exist" containerID="156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e" Oct 30 22:57:43.947808 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:43.947758 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e"} err="failed to get container status \"156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e\": rpc error: code = NotFound desc = could not find container \"156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e\": container with ID starting with 156debcb838491640f7071241bca5059432d030a3a59a3ac5aef7ab71d6eaf1e not found: ID does not exist" Oct 30 22:57:44.096452 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.096424 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5"] Oct 30 22:57:44.100513 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.100480 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-2-w0-d09fff-kserve-76bf45b45c-58zb5"] Oct 30 22:57:44.182758 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.182683 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:57:44.213960 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.213932 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6"] Oct 30 22:57:44.218037 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.218017 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/premium-w0-d09fff-kserve-57455dc7f-jgbg6"] Oct 30 22:57:44.593930 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.593891 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:44.868874 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.868801 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" path="/var/lib/kubelet/pods/5591863e-ed98-405d-b7ce-7c0c9eeb099f/volumes" Oct 30 22:57:44.869231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:44.869218 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfaba7da-6325-41c6-87da-db337fffd973" path="/var/lib/kubelet/pods/dfaba7da-6325-41c6-87da-db337fffd973/volumes" Oct 30 22:57:52.862214 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:57:52.862172 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:57:54.594689 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:54.594650 5607 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:57:55.639053 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:55.639019 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m"] Oct 30 22:57:55.689924 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:55.689895 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d09fff-qd98m"] Oct 30 22:57:56.868454 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:57:56.868424 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a589e9-7a25-4f16-ab8a-a960ef239e3c" path="/var/lib/kubelet/pods/e0a589e9-7a25-4f16-ab8a-a960ef239e3c/volumes" Oct 30 22:58:00.342160 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.342134 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 22:58:00.491229 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.491193 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz\" (UniqueName: \"kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz\") pod \"d4a033be-69cb-42cc-b8aa-833addc63ed7\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " Oct 30 22:58:00.491401 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.491283 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume\") pod \"d4a033be-69cb-42cc-b8aa-833addc63ed7\" (UID: \"d4a033be-69cb-42cc-b8aa-833addc63ed7\") " Oct 30 22:58:00.491673 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.491653 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume" pod "d4a033be-69cb-42cc-b8aa-833addc63ed7" (UID: "d4a033be-69cb-42cc-b8aa-833addc63ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:58:00.493628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.493597 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz" pod "d4a033be-69cb-42cc-b8aa-833addc63ed7" (UID: "d4a033be-69cb-42cc-b8aa-833addc63ed7"). InnerVolumeSpecName "kube-api-access-d7ptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:00.592349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.592324 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7ptz\" (UniqueName: \"kubernetes.io/projected/d4a033be-69cb-42cc-b8aa-833addc63ed7-kube-api-access-d7ptz\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:00.592349 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:00.592349 5607 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a033be-69cb-42cc-b8aa-833addc63ed7-plugins-config-volume\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:01.013488 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.013450 5607 generic.go:356] "Generic (PLEG): container finished" podID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerID="51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44" exitCode=0 Oct 30 22:58:01.013671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.013499 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" Oct 30 22:58:01.013671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.013530 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" event={"ID":"d4a033be-69cb-42cc-b8aa-833addc63ed7","Type":"ContainerDied","Data":"51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44"} Oct 30 22:58:01.013671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.013585 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j" event={"ID":"d4a033be-69cb-42cc-b8aa-833addc63ed7","Type":"ContainerDied","Data":"ecff125edba22728d6f11d1869069cf9f32e6d06c5f72e9dbb0d349d3efab1c9"} Oct 30 22:58:01.013671 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.013610 5607 scope.go:122] "RemoveContainer" containerID="51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44" Oct 30 22:58:01.022144 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.022127 5607 scope.go:122] "RemoveContainer" containerID="51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44" Oct 30 22:58:01.022386 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:58:01.022366 5607 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44\": container with ID starting with 51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44 not found: ID does not exist" containerID="51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44" Oct 30 22:58:01.022451 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.022396 5607 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44"} err="failed to get container status \"51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44\": rpc error: code = NotFound desc = could not find container \"51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44\": container with ID starting with 51cd1b8b631d02c708d11900873ae3dc1e5d1b298f4b42f815dae0f35ea06a44 not found: ID does not exist" Oct 30 22:58:01.185847 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.185827 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j"] Oct 30 22:58:01.236966 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:01.236933 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d09fff-6568ddc9f-jj88j"] Oct 30 22:58:02.868470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:02.868438 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" path="/var/lib/kubelet/pods/d4a033be-69cb-42cc-b8aa-833addc63ed7/volumes" Oct 30 22:58:02.870937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:02.870916 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd"] Oct 30 22:58:02.871124 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:02.871105 5607 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" containerID="cri-o://70b936cc36991bd4e93f61ff5753d9b14a0cb7a861a0058a535f43641c517021" gracePeriod=30 Oct 30 22:58:06.867381 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:06.867341 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dfrsk_100c5340-82b8-4e93-bd2a-10ce866db411/manager/0.log" Oct 30 22:58:07.736094 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:07.736061 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:58:07.985290 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:07.985267 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:58:08.051533 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.051501 5607 generic.go:356] "Generic (PLEG): container finished" podID="2e271101-b9ba-4b33-8862-88373435f991" containerID="70b936cc36991bd4e93f61ff5753d9b14a0cb7a861a0058a535f43641c517021" exitCode=0 Oct 30 22:58:08.051703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.051612 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" event={"ID":"2e271101-b9ba-4b33-8862-88373435f991","Type":"ContainerDied","Data":"70b936cc36991bd4e93f61ff5753d9b14a0cb7a861a0058a535f43641c517021"} Oct 30 22:58:08.051703 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.051669 5607 scope.go:122] "RemoveContainer" containerID="3fd5e1fc3c708d32f91e95624f4d27977d20e57650809759074725a1aaab4b1e" Oct 30 22:58:08.118095 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.118074 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 22:58:08.159628 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159596 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.159769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159666 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.159769 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159738 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.159901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159770 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.159901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159805 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.159901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159836 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.159901 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159868 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.160141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159901 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.160141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159983 5607 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy\") pod \"2e271101-b9ba-4b33-8862-88373435f991\" (UID: \"2e271101-b9ba-4b33-8862-88373435f991\") " Oct 30 22:58:08.160141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.159990 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:08.160141 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.160060 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:08.160345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.160276 5607 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-credential-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.160345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.160294 5607 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-data\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.160345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.160254 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:08.160345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.160331 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:58:08.160345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.160334 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:08.162001 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.161946 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:08.162139 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.162099 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:58:08.162231 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.162213 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "kube-api-access-kf7ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:58:08.162387 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.162367 5607 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy" pod "2e271101-b9ba-4b33-8862-88373435f991" (UID: "2e271101-b9ba-4b33-8862-88373435f991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:58:08.228772 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.228750 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:58:08.261641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261619 5607 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7ls\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-kube-api-access-kf7ls\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.261641 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261640 5607 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-certs\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.261773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261649 5607 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-workload-socket\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.261773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261658 5607 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e271101-b9ba-4b33-8862-88373435f991-istiod-ca-cert\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.261773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261667 5607 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e271101-b9ba-4b33-8862-88373435f991-istio-envoy\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.261773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261675 5607 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e271101-b9ba-4b33-8862-88373435f991-istio-token\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:08.261773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:08.261685 5607 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e271101-b9ba-4b33-8862-88373435f991-istio-podinfo\") on node \"ip-10-0-133-91.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:58:09.057470 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:09.057436 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" event={"ID":"2e271101-b9ba-4b33-8862-88373435f991","Type":"ContainerDied","Data":"98fec42216ee0ecf11909abe13e865c37590bf9ffa7dd044768134d024ce743d"} Oct 30 22:58:09.058059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:09.057474 5607 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd" Oct 30 22:58:09.058059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:09.057492 5607 scope.go:122] "RemoveContainer" containerID="70b936cc36991bd4e93f61ff5753d9b14a0cb7a861a0058a535f43641c517021" Oct 30 22:58:09.277078 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:09.277037 5607 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd"] Oct 30 22:58:09.328220 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:09.328192 5607 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-gjcrd"] Oct 30 22:58:10.872178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:10.870533 5607 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e271101-b9ba-4b33-8862-88373435f991" path="/var/lib/kubelet/pods/2e271101-b9ba-4b33-8862-88373435f991/volumes" Oct 30 22:58:21.187770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:21.187737 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:58:21.549937 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:21.549859 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:58:21.780234 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:21.780204 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:58:22.863054 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:58:22.863012 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:58:25.452155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:25.452127 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-91bcc99f-8fl7p_509b73b9-8697-4da1-91a7-50cb970c8184/revoke-keys/0.log" Oct 30 22:58:26.376623 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:26.376592 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-gvp8q_59bfd83b-c3e4-4e10-b631-b12d8ffbf728/postgres/0.log" Oct 30 22:58:30.004164 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:30.004133 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dfrsk_100c5340-82b8-4e93-bd2a-10ce866db411/manager/0.log" Oct 30 22:58:31.410943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:31.410907 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d969c5987-s82hb_d893d4a3-eb6d-47c3-a12c-10e1e99be3d6/kube-auth-proxy/0.log" Oct 30 22:58:32.309392 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:32.309358 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9xpg2_1ff003b9-d2ce-4ba0-b05a-5cca5c076893/payload-pre-processing/0.log" Oct 30 22:58:32.535151 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:32.535106 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:58:33.643259 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:33.643225 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-g2jpn_ed0dca8e-43d3-4eae-a64e-1638db5fdacf/main/0.log" Oct 30 22:58:34.193566 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:34.193538 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b7l5w_a41e3c61-de7d-40bc-9e0a-1874cf3777c1/main/0.log" Oct 30 22:58:34.418190 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:34.418155 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9mgj6_635e884d-8183-4f26-b773-cc004f23195b/main/0.log" Oct 30 22:58:34.924235 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:34.924206 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-xr5st_6f150d8f-d2b5-4904-b322-c19317461e15/main/0.log" Oct 30 22:58:45.585838 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.585800 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4s68x/must-gather-wpph4"] Oct 30 22:58:45.586266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586234 5607 cpu_manager.go:414] "containerMap: removing container" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerName="payload-pre-processing" Oct 30 22:58:45.586266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586248 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerName="payload-pre-processing" Oct 30 22:58:45.586266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586260 5607 cpu_manager.go:414] "containerMap: removing container" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" Oct 30 22:58:45.586266 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586264 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586272 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586277 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586283 5607 cpu_manager.go:414] "containerMap: removing container" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586289 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586305 5607 cpu_manager.go:414] "containerMap: removing container" podUID="509b73b9-8697-4da1-91a7-50cb970c8184" containerName="revoke-keys" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586309 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509b73b9-8697-4da1-91a7-50cb970c8184" containerName="revoke-keys" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586318 5607 cpu_manager.go:414] "containerMap: removing container" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586322 5607 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586403 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a033be-69cb-42cc-b8aa-833addc63ed7" containerName="payload-pre-processing" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586413 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="5591863e-ed98-405d-b7ce-7c0c9eeb099f" containerName="main" Oct 30 22:58:45.586414 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586418 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" Oct 30 22:58:45.586773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586424 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e271101-b9ba-4b33-8862-88373435f991" containerName="istio-proxy" Oct 30 22:58:45.586773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586431 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="509b73b9-8697-4da1-91a7-50cb970c8184" containerName="revoke-keys" Oct 30 22:58:45.586773 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.586436 5607 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfaba7da-6325-41c6-87da-db337fffd973" containerName="main" Oct 30 22:58:45.589770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.589752 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.648291 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.648259 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4s68x/must-gather-wpph4"] Oct 30 22:58:45.697020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.696979 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4s68x\"/\"default-dockercfg-7g7rc\"" Oct 30 22:58:45.697020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.696989 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4s68x\"/\"kube-root-ca.crt\"" Oct 30 22:58:45.697020 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.696992 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4s68x\"/\"openshift-service-ca.crt\"" Oct 30 22:58:45.711939 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.711914 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2tk\" (UniqueName: \"kubernetes.io/projected/4313dbd1-337d-453c-b698-bc836d86e019-kube-api-access-rb2tk\") pod \"must-gather-wpph4\" (UID: \"4313dbd1-337d-453c-b698-bc836d86e019\") " pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.712054 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.712000 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4313dbd1-337d-453c-b698-bc836d86e019-must-gather-output\") pod \"must-gather-wpph4\" (UID: \"4313dbd1-337d-453c-b698-bc836d86e019\") " pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.813348 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.813312 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4313dbd1-337d-453c-b698-bc836d86e019-must-gather-output\") pod \"must-gather-wpph4\" (UID: \"4313dbd1-337d-453c-b698-bc836d86e019\") " pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.813540 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.813378 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2tk\" (UniqueName: \"kubernetes.io/projected/4313dbd1-337d-453c-b698-bc836d86e019-kube-api-access-rb2tk\") pod \"must-gather-wpph4\" (UID: \"4313dbd1-337d-453c-b698-bc836d86e019\") " pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.813668 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.813648 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4313dbd1-337d-453c-b698-bc836d86e019-must-gather-output\") pod \"must-gather-wpph4\" (UID: \"4313dbd1-337d-453c-b698-bc836d86e019\") " pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.875731 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.875662 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2tk\" (UniqueName: \"kubernetes.io/projected/4313dbd1-337d-453c-b698-bc836d86e019-kube-api-access-rb2tk\") pod \"must-gather-wpph4\" (UID: \"4313dbd1-337d-453c-b698-bc836d86e019\") " pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:45.904217 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:45.904190 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4s68x/must-gather-wpph4" Oct 30 22:58:46.274100 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:58:46.274055 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4313dbd1_337d_453c_b698_bc836d86e019.slice/crio-164421bcef15fe232d87b81505dc602544a0d60ce4f43da5206c21737fa63304 WatchSource:0}: Error finding container 164421bcef15fe232d87b81505dc602544a0d60ce4f43da5206c21737fa63304: Status 404 returned error can't find the container with id 164421bcef15fe232d87b81505dc602544a0d60ce4f43da5206c21737fa63304 Oct 30 22:58:46.276171 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:46.276149 5607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:58:46.276244 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:46.276193 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4s68x/must-gather-wpph4"] Oct 30 22:58:47.240711 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:47.240675 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4s68x/must-gather-wpph4" event={"ID":"4313dbd1-337d-453c-b698-bc836d86e019","Type":"ContainerStarted","Data":"164421bcef15fe232d87b81505dc602544a0d60ce4f43da5206c21737fa63304"} Oct 30 22:58:52.862375 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:58:52.862336 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:58:53.278400 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:53.278359 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4s68x/must-gather-wpph4" event={"ID":"4313dbd1-337d-453c-b698-bc836d86e019","Type":"ContainerStarted","Data":"f4bee618661a603d0ba1b1eb45472b2f8469e11b3cf01d8fcc875feeff25b5d3"} Oct 30 22:58:53.278400 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:53.278405 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4s68x/must-gather-wpph4" event={"ID":"4313dbd1-337d-453c-b698-bc836d86e019","Type":"ContainerStarted","Data":"2ae17106da6148d6bd7b853a5cdb5e08e3c4770b589adf63fc7bca0cf340399a"} Oct 30 22:58:53.396038 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:53.395892 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4s68x/must-gather-wpph4" podStartSLOduration=2.048608769 podStartE2EDuration="8.395877149s" totalImagesPullingTime="6.34726838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:53.395681003 +0000 UTC m=+4312.796237137" watchObservedRunningTime="2026-10-30 22:58:53.395877149 +0000 UTC m=+4312.796433270" Oct 30 22:58:55.928780 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:55.928745 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-687l6_1bdaec03-c2f8-446e-9376-bed9a55677c9/global-pull-secret-syncer/0.log" Oct 30 22:58:56.941465 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:56.941435 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:58:57.137596 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:57.137564 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:58:57.374745 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:57.374665 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:58:58.129482 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:58.129448 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:58:58.358680 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:58.358655 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:58:58.609770 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:58.609741 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:58:58.900605 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:58.900530 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dql2n_729a10ce-45d9-4094-8f2b-c5af2b19d159/konnectivity-agent/0.log" Oct 30 22:58:59.781799 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:58:59.781771 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-91.us-west-2.compute.internal_4385e5f7b5de508595fd215a6736486a/haproxy/0.log" Oct 30 22:59:02.665925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:02.665893 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:59:02.864271 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:02.864238 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:59:03.098314 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:03.098282 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:59:04.392575 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:04.392540 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:59:04.587194 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:04.587160 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:59:04.825498 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:04.825465 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:59:07.458203 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.458169 5607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644"] Oct 30 22:59:07.469099 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.469076 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.524717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.524689 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7d8m9\"/\"kube-root-ca.crt\"" Oct 30 22:59:07.524717 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.524727 5607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7d8m9\"/\"default-dockercfg-qb7pg\"" Oct 30 22:59:07.533783 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.533758 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-podres\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.533973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.533787 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-proc\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.533973 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.533831 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-lib-modules\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.534080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.533964 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgwp\" (UniqueName: \"kubernetes.io/projected/1f279fe9-6f59-4358-a842-aae41077e032-kube-api-access-6xgwp\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.534080 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.534033 5607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-sys\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.575781 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.575748 5607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7d8m9\"/\"openshift-service-ca.crt\"" Oct 30 22:59:07.577466 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.577443 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644"] Oct 30 22:59:07.635486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635450 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-podres\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635486 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635489 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-proc\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635539 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-lib-modules\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635582 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-podres\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635588 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgwp\" (UniqueName: \"kubernetes.io/projected/1f279fe9-6f59-4358-a842-aae41077e032-kube-api-access-6xgwp\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635600 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-proc\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635663 5607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-sys\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635735 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635673 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-lib-modules\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.635988 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.635740 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f279fe9-6f59-4358-a842-aae41077e032-sys\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.700350 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.700319 5607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgwp\" (UniqueName: \"kubernetes.io/projected/1f279fe9-6f59-4358-a842-aae41077e032-kube-api-access-6xgwp\") pod \"perf-node-gather-daemonset-q7644\" (UID: \"1f279fe9-6f59-4358-a842-aae41077e032\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:07.782178 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:07.782106 5607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:08.153829 ip-10-0-133-91 kubenswrapper[5607]: W1030 22:59:08.153791 5607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f279fe9_6f59_4358_a842_aae41077e032.slice/crio-193c3d191da4447e553893aa1505ba389c3f28f74dbef7cb4c5e158990d7a181 WatchSource:0}: Error finding container 193c3d191da4447e553893aa1505ba389c3f28f74dbef7cb4c5e158990d7a181: Status 404 returned error can't find the container with id 193c3d191da4447e553893aa1505ba389c3f28f74dbef7cb4c5e158990d7a181 Oct 30 22:59:08.154059 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.154033 5607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644"] Oct 30 22:59:08.347322 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.347291 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" event={"ID":"1f279fe9-6f59-4358-a842-aae41077e032","Type":"ContainerStarted","Data":"e610bf3bbca48cc882b67b6fb13e9c0e2102d99441d6b397cf2e0c1c2b3fabd4"} Oct 30 22:59:08.347490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.347340 5607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" event={"ID":"1f279fe9-6f59-4358-a842-aae41077e032","Type":"ContainerStarted","Data":"193c3d191da4447e553893aa1505ba389c3f28f74dbef7cb4c5e158990d7a181"} Oct 30 22:59:08.347490 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.347371 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:08.466198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.466172 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:59:08.466198 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.466165 5607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" podStartSLOduration=1.466150723 podStartE2EDuration="1.466150723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:08.464635812 +0000 UTC m=+4327.865191934" watchObservedRunningTime="2026-10-30 22:59:08.466150723 +0000 UTC m=+4327.866706884" Oct 30 22:59:08.662105 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.662077 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:59:08.691304 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.691270 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dfrsk_100c5340-82b8-4e93-bd2a-10ce866db411/manager/0.log" Oct 30 22:59:08.895943 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:08.895844 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:59:10.360669 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:10.360582 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9ddfdbfbf-zn9hh_8eb14762-8c2d-40da-91a6-0141b812fb3c/manager/0.log" Oct 30 22:59:10.548340 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:10.548310 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwlj6_7308c1fa-b891-4a24-b173-cd6a9d361851/manager/0.log" Oct 30 22:59:10.784055 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:10.784022 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-rp479_525b6cdb-6e46-470f-b8a6-9fde60d3cb3f/manager/0.log" Oct 30 22:59:14.364591 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:14.364561 5607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-q7644" Oct 30 22:59:15.513183 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:15.513142 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d969c5987-s82hb_d893d4a3-eb6d-47c3-a12c-10e1e99be3d6/kube-auth-proxy/0.log" Oct 30 22:59:15.930691 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:15.930606 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fd6cdc7bc-bdwzf_bb95c8ea-e1a6-40f5-963f-19005925a1f8/metrics-server/0.log" Oct 30 22:59:16.335975 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:16.335925 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9xpg2_1ff003b9-d2ce-4ba0-b05a-5cca5c076893/payload-pre-processing/0.log" Oct 30 22:59:16.534841 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:16.534817 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:59:17.358092 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:17.358066 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d969c5987-s82hb_d893d4a3-eb6d-47c3-a12c-10e1e99be3d6/kube-auth-proxy/0.log" Oct 30 22:59:18.202265 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:18.202232 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9xpg2_1ff003b9-d2ce-4ba0-b05a-5cca5c076893/payload-pre-processing/0.log" Oct 30 22:59:18.414748 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:18.414720 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8cgnk_2972d9e0-bf8e-4dc6-b7b8-25caee856f57/payload-processing/0.log" Oct 30 22:59:19.226233 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:19.226193 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fq2xz_ab980c40-b10c-4f42-90d8-bd930becdb72/node-exporter/0.log" Oct 30 22:59:19.619925 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:19.619817 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fq2xz_ab980c40-b10c-4f42-90d8-bd930becdb72/kube-rbac-proxy/0.log" Oct 30 22:59:20.018333 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:20.018309 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fq2xz_ab980c40-b10c-4f42-90d8-bd930becdb72/init-textfile/0.log" Oct 30 22:59:20.421865 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:20.421775 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-9gxsh_ffe895c1-e716-45d5-94eb-58668c4bd1c1/kube-rbac-proxy-main/0.log" Oct 30 22:59:20.823185 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:20.823150 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-9gxsh_ffe895c1-e716-45d5-94eb-58668c4bd1c1/kube-rbac-proxy-self/0.log" Oct 30 22:59:21.216599 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:21.216569 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-9gxsh_ffe895c1-e716-45d5-94eb-58668c4bd1c1/openshift-state-metrics/0.log" Oct 30 22:59:21.629768 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:21.629698 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/prometheus/0.log" Oct 30 22:59:22.021345 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:22.021318 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/config-reloader/0.log" Oct 30 22:59:22.414015 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:22.413917 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/thanos-sidecar/0.log" Oct 30 22:59:22.810850 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:22.810823 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/kube-rbac-proxy-web/0.log" Oct 30 22:59:22.862030 ip-10-0-133-91 kubenswrapper[5607]: E1030 22:59:22.861994 5607 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phd9k" containerName="extractor" Oct 30 22:59:23.206682 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:23.206656 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/kube-rbac-proxy/0.log" Oct 30 22:59:23.600066 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:23.599939 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/kube-rbac-proxy-thanos/0.log" Oct 30 22:59:23.996026 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:23.995989 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8a14e049-63a2-40ed-922d-cf72de38a167/init-config-reloader/0.log" Oct 30 22:59:24.070155 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:24.070126 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dfrsk_100c5340-82b8-4e93-bd2a-10ce866db411/manager/0.log" Oct 30 22:59:25.295588 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:25.295540 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-tgqt4_6f4e46f2-7b2a-4518-861c-fb94540b0231/prometheus-operator-admission-webhook/0.log" Oct 30 22:59:26.735382 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:26.735355 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dfrsk_100c5340-82b8-4e93-bd2a-10ce866db411/manager/0.log" Oct 30 22:59:26.940534 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:26.940485 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b796bff-x4mrn_3ebfccd4-cbd5-4319-be94-3da3df1d8566/thanos-query/0.log" Oct 30 22:59:27.333430 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:27.333382 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b796bff-x4mrn_3ebfccd4-cbd5-4319-be94-3da3df1d8566/kube-rbac-proxy-web/0.log" Oct 30 22:59:27.725721 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:27.725699 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b796bff-x4mrn_3ebfccd4-cbd5-4319-be94-3da3df1d8566/kube-rbac-proxy/0.log" Oct 30 22:59:28.119135 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:28.119051 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b796bff-x4mrn_3ebfccd4-cbd5-4319-be94-3da3df1d8566/prom-label-proxy/0.log" Oct 30 22:59:28.513443 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:28.513392 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b796bff-x4mrn_3ebfccd4-cbd5-4319-be94-3da3df1d8566/kube-rbac-proxy-rules/0.log" Oct 30 22:59:28.906432 ip-10-0-133-91 kubenswrapper[5607]: I1030 22:59:28.906347 5607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b796bff-x4mrn_3ebfccd4-cbd5-4319-be94-3da3df1d8566/kube-rbac-proxy-metrics/0.log"